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EXECUTIVE SUMMARY AND RECOMMENDATIONS 

Background 

The Basic Educational Opportunity Grant (BEOG) is the largest of the 
student financial aid programs administered by the Office of Student 
Financial Assistance (OSFA). BEOG was authorized by Title IV of the 
Higher Education Act of 1965, and is second only to the Guaranteed 
Student Loan program in total student compensation. Students who receive 
SEOGs are also eligible for other types of State and Federal financial 
aid; thus,. the BEOG serves as a cornerstone of aid, to students who are 
eligible based on a,formu-la which determines financial need. The result 
of this formula calculation is a student eligibility index (SEI) which, ■ 
together with cost of education at the institution .the student plans to 
attend, and the student's enrollment status (full time or part time), 
determines, the amount of the 3E06 to which the student is entitled. 

One of the OSFA's management objectives is to reduce the amount of 
student misreporting on SEOG applications, or to increase the rate of 
awards based on accurate information. To meet this objective, OSFA has 
instituted several procedures for detecting and correcting errors on 
applications; namely, application processing system- edits and prs-award 
validation of selected applicants by financial aid administrators 
(FAAs). The purposes of this study are to evaluate the impact of these 
quality assurance procedures on the correct award of BEOGs,. and to 
develop a model for detecting error-prone applications early in their 
processing. 
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Description of the Applica tion Processinf| FHitc 

Students may apply for a BEOG using one of several application 
forms. All applications are, however, sent to a central processing 
agency, where a student eligibility report (SER) results and is mailed to 
each applicant. If all application information appears complete and 
consistent, an SEI is computed. To be eligible for a grant, the value of 
the SEI must be 1600 or ^ess. For eligible applicants, a maximum 
potential award (based on full time enrollment at the institution 
indicated as the applicant's first choice) is calculated. Both the SEI 
and award amount are printed on the SER which the student brings to the 
^AA for final award calculation based on actual enrollment status. 

In many cases, however, the information initially reported on thp 
application appears inconsistent or incomplete. This determination is 
maae by the edits, which are applied to all applications as they aro 
processed. In 1979-80. nearly half (42^) of the 3.966,448 applicants 
received an edit to an application item which was critical to calculation 
or the SEL These edits result in comments which are printed on the SER 
and returned to the student for review, corrections or verification 
Many of tnese coirments cause the. application to be rejected; that is. an 
SEI will not be computed unless the student responds to the comment by 
verifying or correcting the items indicated. In this academic year. 36 
percent of all applicants received rejection comments. The following • 
chart shows the rejection and eligibility status of Basic Grant 
applicants. 
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Descripti on of the Pre-Award Institution Validation Procoss 

In academic year 1973-79, BSFA initiated a procedure for validation 
of.certain data items on selected applications by financial aid 
administrators at the institutions to whith these students apply. (The 
chart on the following page shows the distirubtion of Validation and 
nonvalidation, on applicants for the 1979 •30 academic year). BSFA had 
developed criteria, which according to several previous studies, 
indicated inaccurate reporting on applications. These preestabl ished 
criteria (PEC) have been refined and were used to select 166, 348 
applicants for validation. In addition , a smaller group of applicants 
were randomly-seleted to be validated. This random group was selected 
for comparison with the group selected according to the PEC, as an 
ongoing check on th« effectiveness of the PEC and of the validation 
process. 

The student is informed by a comment on the SER that he/she' has been 
selected for validation, and is instructed to bring documentati.on of 
certain information supplied on the application to' the FAA. This 
documentation and the application are reviewed by the FAA, and the 
student is instructed whether to verify or correct the items in question. 
Summary 

This study has two major thrusts; one, to assess the impact and 
effectiveness of validation and the edits; and two, to explore the 
viability of a statistical sequential search technique in improving upon 
current methods used to identify error-prone applications. The major 
study findings follow^ 

• IMPACT OF PROCESSING SYSTEM EDITS (Chapter 2) 

• The processing system edits had a substantial impact on the 
corrections behavior of applicants. About 42 percent of all 
applicants received at least one edit addressing a key 
application field. About one-half of the applicants responded 
to the edits by correcting the information reported originally. 
However, nearly half of these corrections were inconsequential 
in terms of award potential . 
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DISTRIBUTION OF VALIDATION AND NONVALIDATION APPLICANTS 



TOTAL APPLICANTS 
100% 



TOTAL VALIDATION 
APPLICANTS 

• ■ 7.6% . 



SELECTED 

FOR 
MEETING 

PEC 



SELECTED 
RANDOMLY 



22%-^ 



J 



TOTAL NONVALIDATION 
APPLICANTS 

92:5?^ 



MEETING 
PEC . 



NOT 
MEETING 
PEC 



• The edits impeded many applicants from receiving a grant. The 
rnajor-ty of eligible applicants who had never received an edit 
had become recipients by May, 1980. In contrast, most applicants 
who received an edit were either still negotiating the system or 
had been deterred. Since this study is limited to analysis of 
exisuing data, it is not feasible to determine the reason for 
the lower rate at which these applicants attained recipient 
status. 

• About 36 percent of all applicants were prevented from receiving 
an eligibility determination by the edit system for incomplete 
?non'"'?o®"^^-^ inaccurate application information. As of May, 
1980, 78 percent of these applicants .had re-entered the system 
and received an eligibility determination. 

• Based upon the results of institution validation, the edit 
system appeared to identify inaccurate and incomplete data. For 
van elation applicants, a majority of whom corrected in response 
to an edit prior to selection for validation, Jth£_gdiis_yere so 
eftective in soliciting valid information thatjthey rendered 
validation unnecessary for .approximately 85 percent"~of "the 
^ses. This rinding assumes that institutional Val i'dati'on 
results in 'valid" information, which is an untested assumption. 

• Edits associated with the Social Security and Veteran's Benefits 
tape matches appear to be the most effective of all edits in 
Identifying inaccurate application information; the edit given 
uo applicants reporting a very low income appears to'TF'the " 
'^i!lA'f55*.L^lJr'. i^en.Vi^/.i'"9 invalid data. ~ . 

VALIDATION (Chapters 3 and 4) 

f Selecting applicants for validation based on the preestablished 
criteria is in general, a more effective approach to selecting 
applicants for validation than the random selection process. A 
greater percentage of PEC applicants than random applicants 
correct post-selection. These corrections result in a larger 
average positive SEI change, hence greater savings to the 
Department o; Education. 

• ^Q_aroyPXA._ard. D - groups based not on the applicants oast 
corrections history- but instead on information on the apislicants 
current financial status - are the best predictors of 
misraporting. Groups _B. a.nd"C - groups Wsed' on 'correction 
history - are no more effective than. random selection' in 

• identifying applKantTni1<ely to correcr'post^sri^crron. 

• There is great variation in the efficacy of the PEC subgroups. 
Seven of the PEC subgroups perform exceptional Iv well in 
Identifying misreporters (A-5,. A-5, A-10, A-U,"A-14, A-15 and 
B-j;. These subgroups elicit high numbers of applicants 
correcting, have relatively large SEI changes, and are 
discriminating in selecting only those applicants whose 
corrections will affect their potential award. 
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• PEC sjubjroups A-1 (household size or marital status missing) and 
Offrf income^ppli cants) A-4 continue to be most-ineffective in 
Identifying potential misreporters. Data indicate that a low 
percentage 0f-isTiral1-lTTCt3mB--jp^ correct in response to 

validation and that of the percent correcting, only a very few 
applicants make correction that affects their SEI. These 
applicants remain eligible more often than other groups of 
applicants, yet do not appear on the recipient file as often 
thus suggesting that the current edits and validation system'may 
be unnecessarily hindering them in obtaining an award. The fact 
that .inancial aid administrators may find it difficult to 
validate income sources or the lack of sources particular to 
this subgroup, may mean that, although the data indicate this 
group to be accurate reporters, in reality the applicants may be 
submitting inaccurate information' that is not being detected by 
current validation procedures. 

ERROR PRONE MODEL (Chapters 5-8) 

The application of a sequential search technique to the randomly 
selected validation simple has shown to be promising in identifying 
error-prone applicants and segregating them into identifiable groups. 
For the purpose of this analysis, "error-prone" applicants are defined as 
those who make corrections resulting in an eligibility index change . 
greater - han 30 points after being selected for validation. The results" 
of this, ort follow. 

• Thirty-seven groups have been identified. They differ from each 
other in the proportion of applicants who: 

do not (seriously) mis report 
misreport to' their advantage 
misreport to their disadvantage 

fail to re-enter the system after selection for validation 

• Within the 37 groups, eight groups account for 28.7 percent of 
all misreporting to the applicant's advantage. Applicants 
belonging to these groups make up eleven percent of the eliqible 
population. ■ 

t In contrast, the pre-established criteria currently used by OSFA 
to select applicants for validation account for 5.9 percent of 
the eligibles and locates, only 9.1 percent of the applicants who 
misreport to their advantage. 

f Validating applicants in these eight groups would be far 

superior to randomly selecting applicants for validation and 
selecting applicants according to the pre-established criteria. 
A total of 45.5 percent of the applicants in these groups 
increased their eligibility index by 50 or more points after 
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validation. • Only 20.3 percent of the randonly selected 
validation applicants and 29.2 percent of the applicants 
selected according to tha pre-established crieteria made this 
type of change. 

Recomnendations_ 

Based upon these study findings, the following recommendations are 
offered to OSFA: 



RECOMMENDATION ffl: Conduct a field study to determine whether 

institutions conduct validation. 
The major findings in this report concerning impact are based on the 
assumption that institutions conduct validation according to OSFA - 
prescribed procedures and that validation results in correct 
information. This is an untested assumption, and a fiald study 
should be conducted to determine whether it is a valid assumption. 

RECOMMENDATION #2: Conduct a field study to determine why 

corrections are made. 

The current system requires that other key assumptions be made, in 
particular that corrections made after receipt of an edit or after 
selection for validation are being made in response to the systen. 
The degree to which system actions cause the corrections versus 
corrections being made because applicants are confused or are trying 
to test the sysi-em or for some other reason are unknown. A fipld 
study needs to be conducted to determine the extent to which 
corrections mean what they are assumed to iSean 

RECOMMENDATION #3: Conduct a field study to find out why applicants 

drop out of the system after receiving an edit or 
validation. 

Applicants who receive an edit or are selected for validation and who 
subsequently fail to re-enter the systen or to obtain an award after 
negotiating the system are a mystery. It is hypothesized that some 
applicants, do not return because they are misreporting and have been 
i Jccf: "^'^ ^^^^''^ ^""^ frustrated and have. given up. In order 
tor OSFA to make approp irate decisions about the edits and 
validation a field study needs to be done to' find out why these 
applicants drop out of the system. • ""J^. ^nese 

• RECOfflENDATION #4: Improve timeiness in obtaining award data. 
The dollar impact of the edits and validation is key in providing a 
basis for sound decisionmaking. However, there is a significant time 
lag between when the student receives an award and when OSFA obtains 
informtion on the award amount. OSFA needs to review its financial 
system and make improvements to obtain timely information on award 
expenditures. 

Within the context of the current system and the data which were 
available for this study, the following recommendations are made: 
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RECOMMENDATION #5: Unless OSFA alters the current edit and 

cJi i2 ^ysts"'* the edits which focus on lowiiocpine^appiic^nts 

JJ^'^^'^^*! ^^"^ ^° "^22 more^pplicants to d^op out of 
J2r.?Xf flo^^J-.^" ^"'■t^s^' those applicants who 

to corrections '*e-enter the system show virtually no impact 

RECOMMENDATION #6: Continue to expand and refine the tape match 
5; J -y findings have shown that the matches with SSA and VA 
\" identifying and correcting misreporting than 
other edits OSFA should refine its current efforts hy studying "ays 
to improve the current match rate with SSA and VA. Givlen the 
effectiveness of these efforts, OSFA should try to clear legislative 
channels to a low additional tape matches with other dSa ba es! 
particularly the Internal Revenue Service. ^^^'==, 

RECOMMENDATIOM #7: With the. ^eptXojn.^the low income edits. 

-^nnl-lJf ^^^'^''''f^^^IJl^ ^P°" vaTTdaTionTil^HT?iii7- t 

'""'^ edits arF-effective and eliminate the need for 
validation for many applicants. 

RECOf^lMENDATION #8: Revise the validation selection procedures to 
incorporate the findings of the error-prone model (EPM). Preliminary 

luatr'dr '^To'].°^ 9'"°"P^ identified in the £PM are ^ 

superior to the current PEC in identifying misreporting, and that 
they should be used with the successful PEC subgroups for identifying 
error-prone applications. Before using the results of the °PM it ?s 
important that it be updated using more recent applicant and payment 
data than were used in this exploratory study. paynieni: 

tf^2?^!l°JIJ?'^ ^^i W-P^^ -^■'^■^atif ied the estimation of income- and ' 
tAxes„by...app brants. 45ffi3ubnii±. ^p.lx 

t"c.o!!)f4^^..Re|.'Jrn as .a.majqr.^ of error. OSFA rnaV' elTminTte'Vuch 
errors through a range of actions with the followinr^s sSo^^^^ 

• Mot accepting applications prior to the Federal tax filing 
ueadiine 



or 



• Requiring copies of tax returns from all applicants who 
apply before they have completed the Federal tax return. 

?is'?fI;Hl^?'/"i''^ alternative would place a serious burden on * 
Jtf,Hlnt^?«- ^ i packaging schedules, which in turn would affect 
!LnnH °^ school, It is recommended that OSFA entertain the 

thp^oc? ^•''"f'^!* J^^'^ ""^d accomplished through several ways 
the most simple of which would be to either modify processing 

inlStntfon^f"^ I'^V'J ^'^^ "'^^ application o? modify 

;e"l'u;S"Slt°rJni;^;S?nh^r°g5r^%^?^ '° ^'"^^"^^ ^^^'^^^^^^ °^ t- 

nisrS?I^°I! ''^^^'^ '^''^^'^y applicants who are 

^IciS^n^I ^ ^° ^''^r disadvantage, and develop a technical 
assistance program to facilitate their receipt of the correct 
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entitlement. The EPM has identified some groups of students who 
error to their disadvantage. These students should receive edits 
advising them to seek assistance in completing their application, 
either rrom their financial aid administrators or from one of OSFA's 
information service contractors. Applicants should be advised that 
they may be eligible for a larger award to encourage action on their 
part. The Student Eligibility Report should contain a code to cue 
the technical assistance provider to possible nature of the problem. 



19 



STUDY OVERVIEW 



1.1 Purpose and Scope 

For the 1978-79 academic year, the Bureau of Student Financial 
Assistance (BSFA) introduced two major program initiatives intended to 
ensure that applicants were submitting accurate information on their 
Basic Grant application form. The new program procedures tightened 
certain edits in the processing system, and required, financial aid 
administrators from post-secondary educational institutions to validate 
certain application information of .selected students. Last year, the 
BSFA introduced new comments into the edit system and expanded the pre- 
established criteria (PEC) used to select students for validation. Part 
I of this report examines the impact of these changes on the correct 
award of Basic Grants during the 1979-80 academic year. 

Part I of the report is divided into discussions of three major 
issues: the impact of the current processing system edits, the 
effectiveness of the pre-established criteria as indicators of 
misreporting, and the impact of validation on the application 
information. Chapters 3, and 4 of this part provide indepth analysis 
of these factors. The remainder of this chapter discusses the research 
objectives and questions, study design and population, and methodology 
and data caveats associated with each of these issues. 

1.2 Key Study Objectives Questions 

The broad objectives of the Edits and Validation analyses presented 
in this Part are three-fold. They are: 



1.1 



• to asssi-s the impact of the processing system edits on applicant 
oenavior; '^'^ 

• ^^"2^ the adequacy of the pre-established criteria as 
effective indicators of misrep ortingj and 

• aSar'foTja1ic'1rl^?s!' "^^'"^"^^ °" "^^^^ 

The specific study objectives and study question which guided these 
analyses are divided into three sections which correspond to the broad 
study objectives listed above. 

• The Processing System Edits 

- to determine the type and extent of corrections made by 
applicants in response to the edits, and whether their 
corrections behavior differs substantially from that of 

. applicants who were not screened by the edits; 

~ to determine who is being rejected by the processing system 
and, in particular, whether the edits are unnecessarily 
rejecting low income, needy applicants; . 

to determine the extent to' which the rejection edits ar° 
impeding aopli cants from re-entering the system and 
receiving an eligiblity determination; 

?ejitio?l"lnd^° re-entering the system following a 

to determine the relative efficacy of the individual 
rejection edits in identifying incomplete and inaccurate 
applications. 

• The Pre-established Criteria 

- to determine whether the pre-established criteria are more 
effective than the random selection process in identifying 
applicants likely to make Dost-selection corrections 
resulting in significant SEI change; 

to determine .the relative efficacy of the four 
•pre-established criteria groups (A, B, C & D) and subgroups 
in Identifying students reporting incorrect application 
information; 

- to determine the consistency with which the PECcriteria 

'P^'sf'epoi'ters: that is,, the degree to which the 
PEC Identify only those applicants whose corrections result 
in SEI change; 

r- to determine the relationship between the reason a PEC 

applicant was selected for validation (the. PEC met) and the 
critical fields corrected post-selection by the applicant- 
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- to determine whether PEC applicants have greater difficulty 
in achieving recipient status than random applicants; and 

— to determine the potential dollar saving resulting from the 
use of the PEC as selectors for validation. 

• Validation (and the Interaction of Validation and Edits) 

to determine the extent and direction of corrections made 
by validation applicants prior to their selection for 
validation and whether their correction behavior differs 
substantially from that of applicants not selected for 
validation; 

to determine the extent and direction of corrections made 
by validation applicants post-selection and whether their 
correction behavior differs significantly from 
nonvalidation applicants; 

to determine whether validation affects the consistency 
with which post-selection corrections resulting in SEl' 
change are made: that is, whether a higher or lower 
percentage of validation applicants make corrections that 
do not affect their potential award; 

to determine whether the percent of applicants on the 
recipient file is similar for the validation and 
nonvalidation groups; 

to determine the effect of validation oh the si2° of the 
actual Basic Grant award; and . 

todetermine the relative impact of validation and the 
edits on applicant corrections behavior. 

1.3 Study Desig n 

The three major issues discussed in this Part required slightly 
different study designs for analyses. The following summarizes the study 
design for each issue. 

• ^) The Impact of the Current Processing System Edits 

In general , ^the purpo'se of the analyses of the processing system 
edits is to examine the edit's impact on the following areas: 

• the frequency and type of corrections to key application items; 



the ability of applicants to swiftly meet the demands of the 
processing system and receive an eligibility determination. 
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The analysis of the edits has been divided into two sections. The first 
section examines the global Impact of the edits. The corrections 
behavior and current eligibility (rejected, eligible, ineligible) and 
payment (expected to be paid, not expected to be paid) statuses of 
applicants who received edits are compared with those who did not. In 
addition, the frequency, magnitude and direction of corrections in 
response to the edits are compared with the voluntary, student-initiated 
corrections. 

The second section assesses the relative efficacy of the Individual 
rejection and assumption edits. Each rejection edit is assessed by 
examining applicant corrections and the ability of applicants to re-enter 
the system following rejection. Two major assumptions underlie the 
examination of the adequacy of the rejection edits: 

* It ^ -21^ solicits frequent corrections that have more than a 
negligible impact on the applicant's SEI then it is successfully 
Identifying incomplete or Inaccurate application data. 

f In order to be/worthwhile, a high percentage of applicants must 
re-enter the processing system' following the receipt of the edit. 



b) The Adequacy of the Pre-established Criteria 

To determine the adequacy of the pre-established criteria as' 
indicators of misreporting, a sample of individuals selected for. 
validation because they met one or more PEC criterion was compared with a 
sample selected randomly. These groups were then compared on three 
levels. First, the largegroup of all PEC applicants was compared to the 
group of random appl i cants . to determine whether the concept that • certain 
misreporters can be identified by patterns of applicant behavior or 
information reported by the applicant is viable. Mext, the PEC groups 
(A, B, C and D) were compared to each other to ascertain the relative 
effectiveness of each group in detecting misreporters. Finally, each PEC 
criterion or subgroup was analyzed and its performance in identifying 
applicants submitting incorrect information contrasted with the 
performance of the other subgroups. 
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Each of the PEC groups and PEC criterion was analyzed according to 
several measures of effectiveness. The primary measures used for 
analyses were: 

• '.{ the percentage of applicants correcting any field post-selection 

and those correcting only critical fields; and 

• the magnitude of the resulting average SEI change!/. 

The use of these measures is based on the assumption that corrections 
made post-selection, and hence after the student has been validated, 
reflect instances where validation detected inaccurate information on the 
application and the student corrected. It is assumed that the correction 
reflects accurate information. Theoretically at least, the greater the 
percentage of applicants correcting post-selection, the higher the number 
of misreporters identified. The magnitude of the SEI change which 
results from the correction should also reflect the disparity between the 
old, incorrect application information and the new, correct information. 
Therefore, the more inaccurate the initial information or the more 
serious the misreporter, the greater the SEI change resulting from the 
correction.-'' 



-/The average SEI change is derived by summing the SEI change scores ^ 
for applicants raising and lowering their SEI and dividing this figure 
by the number of applicants correcting, including those whose 
corrections did not result in SEI change. 

i/We realize that these assumptions do not always hold true. For 
example, it is plausible that an applicant's initial information was 
correct and that validation elicited an inappropriate change. It is 
also possible that if the initial data were' accurate, the correction 
reflects a legitimate change in the applicant 's status. Furthermore, 
even when the initial information is incorrect, we have no way of 
verifying concretely that the new information is correct and that the 
applicant IS not substituting new falacious information for old. Since 
we also have no guarantee that validation actually took place, or if 
did take place, that the students' documents were checked thoroughly, 
we can only assume that post-selection correction are indicative of 
misreporting. 
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In addition, several other measures were used to assess 
effectiveness. Although these measures are somewhat secondary to the 
above, they allow for finer analyses and Interpretation of the data. 
These measures are as follows: 

• the percentage of applicants correcting post-selection whose 
corrections do not result in SEI change; 

• the percentage of students raising and lowering their SEI and 
the amount raised and lowered; 

• the relationship between the reason an applicant was selected 
for validation and the fields the applicant corrected 
post-selection; 

• therate at. which PEC and randomly selected applicants enter the 
recipient file, and the current eligibility status of the 
applicants; and 

• the average difference in expected award between the transaction 

?!l,n5]^J-^'^^ student was selected for validation and the current 
ti 3n sac ui on • 

c) The. Impact of Validation and the Interaction n f the Edits and 
Va 11 dati on " : — ■ 

The third issue of concern in Part I of this report is the impact of 
validation on the correct award of Basic Grants. For this Part, we 
compared a sanpie of validation applicants (both applicants meeting PEC 
and those randomly-selected) with a similar sample of nonvalidatlon 
applicants. These groups were compared first on their pre-selection 
correction behavior, (including .the percent correcting and the. fields 
changed) to determine whether the two groups were initially similar 
enough to allow comparisons. The post-selection correction behavior and 
the resulting average SEI change for both groups was then compared to 
assess the degree of difference between the two groups and the Impact of 
validation. Finally, payment data was analyzed to provide some 
indication of the actual dollar savings that might result from validation. 

The assumptions underlying these analyses are the same as those for 
the analyses of the effectiveness of the pre-established criteria; that 
1s, post-selection corrections reflect the changing of inaccurate data to 
accurate data and thus reflect misreporting; and the magnitude of the 
average SEI change is indicative of. the degree of misreporting. 
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Although not considered for analytical purposes a separate Issue, we 
also attempted to determine the relative Impact of the edits and 
validation by developing tables which compare the validity of data fields 
at various points in the application processing and validation cycle, 
This allowed us to determine the approximate degree to which validation' 
is necessary to insure accurate information. 

1.4 Study Population 

The population studied for these analyses consisted of 3,966,448 
applicants, or all individuals on the applicant and recipient files as of 
May 1980, except_for a fairly small gro up of indepen dent ^students whose 
data h^ry^cou^ 

"duTTo problems in their corrections history resulting from a 
Congressionally mandated recomputation of their SET (see the methodology 
section of this chapter for further discussion). 

To facilitate timeliness and to minimize cost, a ten perc t (10%) 
sample of this entire 3.9 million applicant data base was ui '>r many 
of the analyses. The sample drawn consists of one of every f > 
validation applicants, and one of every fourteen other applicants. (The 
PEC validation applicants were oversampled in relation to the other 
groups to ensure adequate numbers of subjects in each criterion group.) 
Population samples were then produced by assigning the appropriate weight 
to the sampled groups. The reliability of this sample approaches 100 
percent due to the large sample size. 

Different subsets of the sample were used for various analyses. 
Therefore, the total number of applicants reported in various tables of 
.this report differ. In general, the edits chapter of this part of the 
report is based on analyses of the total number of applicants ever 
receiving comnents, and the total number ever rejected. The population 
totals for these groups follow: 
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PRIMARY GROUPS SAMPLED FOR EDITS ANALYSES 



Total Number of Applicants 
Receiving C oinnents 
1,650,522 



Total Number of Applicants 

Ever Rejected 

1,423,540 



The only sample population used in analyzing the adequacy of the PEC 
subcriterion is the set of all validation applicants. This sample is 
divided into those applicants meeting the pre-established criteria those 
selected randomly!/ The file reports the following distribution of 
validation applicants. 



SAMPLE POPULATION FOR PEC ANALYSES 



Total Number of Validation Applicants 212,362 
Number Meeting PEC 166,348 
Number Randomly Selected 46*014 



Finally, the validation chapter of this Part is based on a sample of 
all validation. and nonvalidation applicants. All applicants included in' 
this sample have, at one point in time been eligible for an award. The 
sample is divided according to whetl- r the applicants are selected for 
validation and according to whether the applicants meet the PEC, or were 
selected randomly (or did not meet PEC). 



i/The randomly" selected group was chosen by flagging every nth applicant, 
it IS not truly random,, however, because the selection ceilings were 
occasional ■ changed by OE, and because those who submitted corrections had 
multiple transactions (and thus a greater chance of selection). All 
applicants, therefore, did not have an equal probability of selection. 
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SAMPLE POPULATION FOR VALIDATION ANALYSES 



Meeting PEC 



TOTAL 



Total 
Applicants 


Column 

n 


Validation 
Aopl 1 cants 


Column 
% 


Nonvalidation 
ApDlicants 


Column 
% 


673,260 


(24) 


166,348 


(78) 


506,912 


(19) 


2.162,926 


(76) 


46,014 


(22) 


2,116.912 


(81) 


2,836,1861/ 


(100) 


212,362 


(100) 


2,623,824 


(lOO) 



1.5 Study Methodology 

The analyses of the current edits, application processing and 
validation system are based on bivariate contingency tables and frequency 
distributions designed and developed specifically for this project. The 
tables analyzed were run on a file which contained merged secondary data 
provided by the Basic Grant central processor and the Oepartment of 
Education. 

It should be mentioned that before any tables were produced a 
subsample of the total population was excluded from the data file. This 
was necessary because during the 1979-80 academic year Congress approved 
a new formula which changed the way financial need was computed for 
independent students. The new formula was put into effect on May 9. 1979 
and consequently the SEI's of all independent students applying before 
that date had to be recomputed. The recomputation changed the SEI's of a 
significant number of independent applicants whose initial- SEI had been 
greater than zero. However, since the new formula was intended .to 
increase aid to independent students, those applicants whose initial, 
pre-May 9, SEI was zero did not have an SEI change. 



i^As mentioned J n the text, all applicants included in the sample used 
for the validation analyses have been eligible for a grant at some point 
in their application history. This narrows down the population from all 
?n^\^?'"f applicants and explains the difference in the total number of 
applicants reported on page 1.8 and this page. 
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Regardless of whether a student's SEI changed as a result of the 
recotnputation, a new SER was generated dll independent applicants who 
filed prior to May 9. The new SER's were only mailed to those students " 
whose SEI had changed. The newly generated SER's, however, were recorded 
as systems generated transactions by the central processor and showed up 
as correction in our data file. 

In order to retain as many independents affected by the formula 
change as possible, yet not confound the data, we ignored the systems 
generated transaction for students whose initial pre-May 9 SEI was zero 
(and thus did not change), and excluded all other affected students from 
the population. Applicants, whose SEI changed as a result of the 
recompute^ could not be included as part of the sample population because 
of the inaiiiliiy to distinguish between SEI changes caused by the 
recompute and changes caused by, other factors such as misreporting. 
1.6 Study Caveats 

As mentioned previously in this chapter, there are several minor 
caveats: that should be considered in reading and using these reports. 
These caveats can be divided into two categories: those relating to the 
assumptions underlying' the use of certain measurement variables, and 
those related to data processing concerns. 

Most of the caveats related. to the assumptions underlying the use of 
certain measurement variables have been discussed earlier^ so they will 
only be surmiarized here. The first caveat concerns the assumption that 
corrections are indicative of misreporting. At this time we do not know 
what percent of all corrections are due to applicants correcting 
inaccurate information. Although most corrections, particularly 
post-selection' validation corrections, would seem to be indicative of 
misreporting, some corrections must reflect changes made for other 
reasons. The Internal Revenue Study currently in progress, will help 
determine the degree to which corrections reflect changes from incorrect 
to correct information. In addition we have no clear evidence whether 
students are accurately validated, or if validated, how thoroughly their 
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documentation is checked; we only know that applicants were selected for 
validation. Before making conclusive statements regarding the impact of 
validation, per se, it will be necessary to determine to a better degree 
the extent and the quality of the validation process. 

There are three other constraints imoosed primarily because of data 
processing reasons that must be considered in using this report. First, 
the data is not representative of the total Base Grant population. As 
mentioned in the methodology' section above, a small, but significant 
percent of the population i.e., (independency students whose SEI was 
increased as a result of the SEI formula change) was excluded from the 
data sample. This means than no inferences can be drawn regarding the 
behavior of this group of applicants. Second, it was necessary to 
compute an artificial student eligibility index for rejected applicants 
in order to conduct the edits analyses. Rejected applicants do not have 
SEI and therefore, the extent to which our assumptions are correct in 
determining the artifical SEI, affects the accuracy of the final data. 
Finally, the reader should be aware that the cerLlu::Al_.^«rc:Hssi3rJs--f^^^ 
to flag nonvalidati on _appli cants meeting PEC groups.J., D, negates 
comparison of scores of the total validation PEC group and the total 
nonvalidatioh PEC group. Only the scores of the A group and subgroups, 
and the random selection (not meeting PEC) groups. can be compared. . 
Otherwise, the exclusion of PEC nonvali dation B, C,- & D applicants will 
resul t rnis leading averages. 
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IMPACT OF THE PROCESSING SYSTEM EDITS AND COMMENTS 



The Basic Grant application processing .system includes several 
features that were designed to minimize the number of student eligibility 
determinations made on the basis of invalid, inaccurate or incomplete 
data. One feature, discussed in detail in Chapters 3 and 4, is the 
selection of applications for validation. Another feature is a series of 
computerized edits which check for missing information and the logic and 
consistency of al 1 .application data provided. 

If an application triggers one of these processing system edits, a 
message is printed on the Student Eligibility Report. (SER) advising the 
student to review the application and take further action, if necessary. 
Under the most restrictive conditions, the edit comment indicates .that 
the application has been rejected and that the student must provide 
missing information or verify or correct existing data before an 
eligibility determination can be made. In other cases, the comment 
indicates that the processing system, in calculating eligibility, assumed 
a value for a missing or apparently Inaccurate application item based on 
other provided information. In the third case, the comment serves as 
merely informational-,, or as an attention-attracting device warning of 
questionable data. 

The purpose of this Chapter is to examine the impact of the 
processing system edits and comments on: 1) the frequency, magnitude and 
direction of applicants' corrections to key application items; and. ?.) 
the ability of applicants to expeditiously- pass through the processing 
system and obtain an eligibility determination. Section 2.1 gives a 
description of the edits' and comments and an overview of the impact of ' 



the edits on applicant behavior. In this section, the corrections 
behavior, the current eligibility status (rejected, eligible, ineligible) 
and payment status (expected to be paid, not expected to be paid) of 
applicants who received edits are compared with those who did not. 
Section 2.2 provides a more detailed analysis of the effect of the most 
restrictive type of edit, the rejection edit. The first part of this 
section examines the income and household characteristics of applicants 
most often rejected and the second part assesses the effectiveness of the 
individual rejection coimients and edits in terms of applicant corrections 
response. 

2.1; tilobal Impact of the Edits 

Description and Distribution of Comments 
There are 317 different computerized comments generated by the 
processing system. For this analysis, ^only the 176 comments that pertain 
to applicants filing a regular application andlYhT^rad"dress"'ke7'^'' " 
application items were examined. (See" Append'ix' A for a list of the 
--asmeoxs used i n IfnTsTuay) . Key application items are defined as those 
which have a major influence in the computation of the Student . 
Eligibility Index (SEI) and/or affect whether an applicant is selected 
for validation. A list of . these key application items follows: 

t Adjusted Gross Income (AGI) 
■ t Taxes Paid (TP) 
• Portions Earned 

Nontaxable Income (Social Security Benefits, Other Nontaxable), 

Veteran's Educational Benefits (VEB) 
Net Assets (NA) ^ 
Applicant's Resources (AR) 

Unusual Expenses (medical/dental, casualty, fire/theft), (UE) 



t 



t Household Size (HS) 



Dependency Status (Model) 

Unreimbursed Elementary and Secondary School Tuition ('jf) 
Post High School Enrollment (PHE) 
Marital Status (MS) 
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t Tax Filing Status (tax return figures are: (a) *rom completed 
return, (b estimated, (c) not appropriate because applicant 
will not file a return), (TFS) t^MMv.ant 

Hereafter, these key data fields are referred to as "critical" fields. 
The abbreviations noted above are used throughout the report. 

As of May, 1980, the 176 coiirients examined in this study had been 
generated 4,164,796 times; 1,650,522, or 42 percent of all applicants, 
had received at least one of these comments, for an average of Z.S 
comments per applicant. Since many applicants receive rejection edits 
prior to being selected for validation, it is not surprising that 
validation applicants received critical field comments at a higher rata 
than no n validation applicants: 84 percent ofvalTdation applicants 
received critical f ie'ld'cbtfrtierrts, for an average of 4.7 per applicant, 
while only 39 percent of all applicants not selected for validation 
received coranents, for an average of 1.3 per applicant. 

As mentioned earlier, there are three types of edits and comments: • 
rejection, assumption and informational. Many applicants received more 
than one type of comnent: , about 65 percent received rejection: comments, 
22 percent received assumption comments and 56 percent received 
informational comments not related to the rejection or assumption edits. 

There was considerable variation by critical field in the number of 
comnents generated. The following table lists the number of applicants 
receiving cotitnents by the critical field which the cormient addressed: (A 
list of the number of applicants who received each individual rejection 
coiirient can be found in Section 1,2.) 

Table 2.1 illustrates that the largest percentages of applicants- 
received corrments to items most critical in determining eligibility. 
This may have occurred because the processing system has the most 
stringent requir^^ments for these fields; it is also indicative of 
questionable data being reported (or not reported) in these, fields. A 
relatively large proportion of applicants received comments addressing 
post high enrollment, a key application item in determining eligibility. 
By comparison, relatively few received comments to household size, an 
equally important field in the award computation.' ■ 
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TABLE 2.1: RATE OF RECEIPT OF COMMENTS BY CRITICAL FIELD 



CRITICAL 
FIELD 

AG I 
TP 

PORTIONS 

NTI 

PHE 

UE 

TFS 

HA 

HS 

VEB 

MS 

AR 

UT 

MODEL 



NUMBER OF APPLICANTS 
RECEIVING COMMENTS 1 / 

1,009,936 
887,825 
709,186 
523,604 
388,122 
214,828 
210,828 
191,796 
179,116 
143,998 
113,316 

73,022 

68,908 

21,384 



PERCENT OF TOTAL 
WHO RECEIVED COMMENTSl / 

61% 

54 

43 

38 

24 

13 

13 

12 

11 

9 

7 

4 

4 

r 



TOTAL APPLICANTS 



1,650,522 



100 



l/Most comnents address more than one field. Consequently the 
rrequencies m this table are duplicated between fields and do not 
.add up to the unduplicated total number of applicants who received 
cntical field comments. For example, an applicant who received a 
comnent which addresses four different fields would appear in four 
of the above frequencies. . 

2/percentages are based on the unduplicated total. number of applicants 
who received critical field comments. 



Since the edits to both household size and post high enrollment are 
equally stringent, this table' suggests that a much higher percentage of 
applicants were apparently misreporting post high enrollment or leaving 
the field blank. 



2.1.2 Corrections In Respons_e to Comnients 

An applicant has the option of changing any application information 
once the SER has been received. A correction may be made at the 
applicant's initiative (instructions on the back of the SER explain how 
to make a correction), it may be made at the direction of the financial 
aid administrator, or it may be made in response to a processing system 
edit. For this study, all changes to critical fields which were in 
response to a comnent are referred to as "solicited" corrections; all 
other corrections are considered "unsolicited". Exhibit 2.1 presents a 
sunrmary of application corrections behavior. 

As the exhibit indicates, about one-fourth of all applicants who were 
ever determined to be eligible corrected a key application field. The 
majority who corrected did so in response to a processing system edit. 
Nearly one half of all^soli cited corrections were inconsequential; that""; 
is, they had no effect on the applicants' SEI. The amount of 
insignificant responses that were elicited suggests that certain 
individual edits may not be cost-effective. By comparison, the SEI 
remained the. same for only one-third of the applicants making unsolicited'' 
corrections. Applicants making unsolicited changes corrected more ■ 
frequently to their advantage ^nd to their disadvantage than applicants 
who responded to the edits. Regardless of whether corrections are 
.solicited or unsolicited, those that have an impact have a greater 
tendency to decrease, rather than increase, award potential. 

To determine further the overall impact of the edits, and to compare 
their influence on applicant corrections behavior with that of pre-award 
validation, the response modes of nonvalidation and validation applicants 
were compared. Table 2.2, which follows, summarizes the frequency and 
magnitude of solicited and unsolicited corrections made by validation and 
nonvalidation applicants both pre- and post-selection/eligibility. As a 
point of reference, post-selection/eligibility corrections occur after 
the transaction when the applicant is selected for validation or after 
the transaction when an initial el igibi 1 ity determination has been made; 
pre-selection/eligibility corrections occur before the applicant has been 
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EXHIBIT 2.1: SUMMARY OF APPLICANT CORRECTIONS 



TOTAL APPLICANTS-/ 
2,836, 186 



APPLICANTS 
NOT CORRECTING 
CRITICAL FIELDS 

2.125.636 

(75%) 



APPLICANTS 
CORRECTING 
CRITICAL FIELDS 

710,550 

(25%) 



CORREC- 
TED TO 
THEIR 
ADVAN- 
. TAGE 

95.146 

(16:i) 



APPLICANTS MAKING 
SOLICITED 
CORRECTIONS 

. 580,366^/ 

(82%) 



NO SEI 
CHANGE 

266 .662 

(46f5) 



CORREC- 
TED TO 
THEIR 
DISAD- 
VANTAGE 

218.558 

(38%) 



APPLICANTS MAKING 
UNSOLICITED 
CORRECTIONS 

303,524^/ 
(43%) . 



CORREC 
TED TO 
THEIR 
ADVAN- 
TAGE 

77 .562 

(26%) 



NO SEI 
CHANGE 

100.852 

(33%) 



CORREC- 
TED TO' 
THEIR 
DISAD- 
VANTAGE 

125,110 

(41:0 



^^Refers to the total number of applicants who ever received an eligible 
-Note thai many appl i cants 'made both solicited and unsolicited corrections' 
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selected or given an initial eligibility determination. The average 
overall effective SEI change is calculated only for those correcting, 
from the time of correction to the most recent transaction. Effective 
change refers to the actual point change if eligibility Indexes above 
1600 (the maximum eligible SEI) are set at 1600. For example, an 
absolute change for an applicant whose SEI goes from 1000 to 2200 is 
1200; the effective change is 600, or 1600 minus 1000. ■ 

As Table 2.2 indicates, most corrections to critical fields occur 
prior to selection for validation or the determination of eligibility. 
The impact of the edits on Scl is also greatest during this time. The 
majority of pre-selection/eligfbi lity corrections were solicited by 
conments whereas more post-selection/eligibility changes were unsolicited. 

As mentioned earlier, validation applicants, receive comments at a 
greater rate than nonvali dation applicants, they also respond to the 
edits more frequently: over half of all validation applicants corrected 
in response to the edits prior to selection while fewer than 20 percent 
of nonvali dation applicants made pre-eligibi lity changes. The data 
suggest that the edits have a greater impact on the validation 
■aDplicant;s corrections behavior than institution validation. Validation 
applicants made notably fewer unsolicited changes after' selection (the 
assumption is that these changes are in response to institution 
validation) than pre-selection corrections in response to the edits. 
Furthermore, solicited post-selection corrections, although .less 
frequent, had an equal effect on applicants' SEI as unsoli cited . 
post-selection corrections. (See Chapter 4 for a detailed examination of 
the interaction of effects of validation and the edits on validation 
applicants.) 

In general, both the validation and nonvalidation applicants who met 
the pre-established criteria (PEC) made more frequent solicited 
corrections that resulted in larger SEI increases than applicants who did 
not meet the PEC. This gives preliminary indication that the rejection 
edits which correspond with the PEC might be working effectively. 
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Table 2.3, which follows, presents the corrections behavior 1n 
response to the edits by the critical field corrected. Once again, the 
population has been divided Into nonv.^ MdaMr- /md validation 
applicants. There were no not. :!- : dan.)-'^ 'n the fields most frequently 
corrected by each applicant group pre- or post-select1on/el1g1b1l1ty. 
The rate of corrections by field can be suimiarized by the following three 
groups: 



Frequently 


Occasionally 


Rarely 


Corrected 


Corrected 


Corrected 


AG I 


TFS 


MS 


NTI 


HS 


VEB 


TP 


UE 


MODEL 


PORTIOMS 


NA 


AR 




PHE 


UT 



As previously mentioned, the majority of corrections, particularly 
pre-selection/sligibility, are in response to a processing system edit. 
However, there is considerable variation by critical field in the rate at 
which applicants make solicited and unsolicited corrections. 

More than 35 percent of the changes to AGI, TP, and VEB followed the 
receipt of an edit, whereas fewer than one half the corrections to NA, 
HS, AR, MS, and Model were solicited. HS, AR, MS, and Model are a few of 
the fields which are supposed to be updated to reflect a change in 
circumstances followinr the original application submission. This may 
explain why a comparatively large number of applicants made unsolicited 
corrections to these fields. 

flonvalidation applicants, in general, corrected all fields to their 
disadvantage pre-el igibil ity, with the exception of HS and MS. This, 
together with the fact that fewer than 40 percent of the corrections to 
HS and MS were in response to comments, suggests that new or more ^ 
restrictive edits might be needed for these two fields. The adequacy of 
the individual edits which address HS and MS is discussed in detail in 
Section 2.2.. 
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All applicants tended to change AGI and TP to their disadvantage both 
pre- and post-selection/eligibility. Validation applicants made large 
corrections to their advantage to nearly all fields prior to selection; 
however, changes to AGI and TP, 90 percent of which came in response to 
edits, were to the applicants' disadvantage. The data strongly suggest 
that the numerous and restrictive edits which address AGI and TP are 
working effectively. 

2-1.3 Current Eli gibility and Payment Status of Ap plicants Receivino 
Conments ' • 

There is a concern that the processing system edits form needless 
roadblocks for many accurate and honest applicants. It is possible that 
some edits are too restrictive or too rigid to make allowances for 
special circumstances that do not meet the norm. Many applicants, 
discouraged or confused about the demands of the processing system, may 
be dropping out and never receiving a grant. 

Exhibit 2.2 compares the current payment status of applicants who 
received conments addressing critical fields with those who did not. As 
a point of reference, applicants who are "expected to be paid" were on 
the recipient file as of May, 1980. 

Exhibit 2.2 shows that the majority of applicants who received 
comments were not expected to be paid, while most applicants who did not 
receive a coimient to a critical field were on the recipient file as of 
May, 1980. Given the May, 1980 deadline for making application 
corrections (with a much later deadline for validation applicants' 
corrections), it is possible that a certain number of those who are 
either currently rejected or ineligible and not expected to be paid will 
re-enter the processing system and receive a grant. Also, some who are 
currently eligible for a grant but are not on the recipient file may 
eventually receive a grant, owing to the late May and early June 
deadlines for applicants to submit their SERs for payment. However, it 
is equally plausible that many applicants in this comparatively large 
group— those receiving comments and not expected to be paid— have dropped 
out and will not receive a grant. 
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EXHIBIT 2.2: WENT P/IYMENT AND ELllillJILITy STATUS: APPLICANTS 

RECEIVING COMMENTS VS. APPLICANTS NOT RECEIVING COfilENTS 
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The following table presents the current payment status of applicants 
who received comnents by the field that the comment addressed. 

TABLE 2.4: CURRENT PAYMENT AND ELIGIBILITY STATUS 3Y CRITICAL FIELD OF 

COMMENT RECEIVED '^iliv,ML ricuu ur 



CRITICAL 
FIELD: 


WHO RECEIVED 
COMMENTS TO FIELD 


% 

EXPECTED TO 
BE PAID 


NOT EXPECTED TO BE 


PflTD 


0/ 

REJECTED 


INEliGIBLE 


% 

ELIGIBLE 


AG I 


1 nno Q'^f^ 


44.1% 


19. -1% 


11.0% 


25.3% 


TP 


PR 7 PP^^ 
OO / ) oco 


42. 6 


21.2 


10.7 


25.4 


NTI 




42.3 


20.3 


10.2 


26.7 


PORTIONS 


700 1 P^; 


41 . 2 


21.6 


10.9 


25.3 


V uu 


"] A'i QQO 

i^oj yyo 


39.0 


12.9 


20.7 


27.4 


HS 


i/^,xXo 


42. 8 


14.9 


12.9 


29.4 


PHF 


388 ,122 


41.1 


14.0 


16.2 


28.7 


NA 


191,796 


48.0 


11.3 


20.9 . 


19.7 


UE 


214,328 


43.4 


18.8 


13.5 


24.2 


UT 


68,908 


39.9 


22.0 


13.8 


24.3 


AR 


73,022 


48.7 


6.9 


27.5 


16.9 


MS . 


U3,316 


39.7 


21.3 


13.1 


25.8 


MODEL 


21,384 


37.9 


2S;6 


8.5 


25.0 


TFS • 


■210,828 


39.9 


25.0 


6.9 


27.2 



As Table 2.4 indicates, there is remarkably little variation in the 
current status of applicants by critical- field cormient received. 
Regardless of the field addressed by the conment, fewer than 50 percent 
of all applicants were expected to be paid and about 25 percent were 
eligible but had not received a grant. However, the current status of 
applicants receiving comnents to NA, AR, TFS, and Model differs 
somewhat. Applicants receiving comments which addressed NA and AR were 
nost likely to appear on the recipient file. -The majority of those not 
2xpected to be paid who received comnents to these two fields were 
:urrently ineligible; relatively few were currently rejected. This 
suggests that the edits and comnents associated with these two fields are 
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not presenting any substantial barriers or discouraging applicants from 
ever receiving a grant. On the other hand, applicants receiving comments 
which address Model and TFS appear to be having trouble negotiating the 
systam. Comparatively few of these applicants are expected to be paid, 
while over 25 percent are currently rejected. 
2.1.4 Suntnary and Conclusions 

The following summarize the key findings regarding the overall impact 
and performance of the processing system edits: 



About 42 percent of all applicants received at least one comment 
addressing a critical application item. Most of the comments 
generated addressed the following four key income fields: AGI. 
TP, Portions and NTI. 

One fourth of all applicants corrected a critical field The 
majority of corrections were in response to processing system 
edits. Nearly one half of the solicited corrections did not 
cause a change in the applicants' SEI. Most of the corrections 
that changed the SEI were to the applicants' disadvantage.- 

The processing system edits have a greater impact on aopli cants 
se ected for validation than on applicants not selected, 
validation applicants received comments at a greater rate than 
nonvalidation applicants. Plus, over 50 percent of validation 
applicants made pre-selection changes in response to the edits, 
whereas only 17. per* cent of nonvalidation. applicants responded to 
the. edits prior to attaining eligibility. It also aopears that 
the edits have agrsate" Impact than institution validation on 
the corrections behavf of validation applicants: ■ only 
one-fourth made unsol..,ted post-selection corrections while 
over one-half . corrected in response to the edits prior to 
selection. 

The following four key income fields were corrected- most often: 
AGI, TP, Portions, and NTI. Least often corrected were: MS 
VEB, Model, AR and UT. Over 85 percent of the corrections to 
AGI and TP were in response to a comment. This, together with 
the fact that corrections to AGI and TP tended to result in 
relatively large SEI changes to the applicants' disadvantaoe, 
indicate that the edits to these two fields are having a ' 
.substantial impact on applicant corrections behavior. 

As of May, 1980, more than half of all applicants who had 
received comments were not expected to be paid, whereas the vast 
majority of those not receiving comments were expected to be 
paid. Comnents relating to TFS and Model seemed to be causing 
applicants particular difficulty. 
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2.2; Rejection Edits and Comments 

The most restrictive of the application processing edits are the 
rejection edits. When an application triggers a rejection edit, a 
conrment is printed on the SER. In order to re-anter the processing 
system and receive an eligibility index, the rejected applicant must 
respond to the cormient by either confirming that the item in question is 
correct or by correcting that item. There are 43 rejection edits, or 
rejection reasons, for applicants filing a regular application. Many 
rejection edits have more than one corresponding comment. The texts of 
the series of rejection comments that correspond to the same rejection 
reason are tailored to the applicant's marital and dependency status. 
For example, conments 10 and 12 are generated for the same reason, but to 
different populations: 10 to dependents with married parents and 12 to 
single independent applicants. This section analyzes the impact and the 
adequacy of the rejection edits and comments. Subsection 2.2.1 examines 
the income and household characteristics most affected by the rejection 
edits and comments, while Subsection 2.2.2 as?i?sses the adequacy of the 
individual rejection edits and comments. 

2.2.1 Characteristics of Applicants Most Frequently Rejected 

As mentioned in Section 2.1* there Is concern that many qualified and 
accurate filers are being impeded' from receiving a grant because (1) the ' 
demands of the edits— the rejection adits in particular— are too' severe 
and (2) the corrections process itself— that is, instruction given by the 
comnents— is confusing. Exhibit 2.3, which, follows , sumnarizes the 
impact of • the rejection edits on the entire applicant population and 
shows the current eligibility status of applicants who had been rejected 
at least p.nce. 

• The exhibit shows that over one-third of all applicants have, at one 
time, received a rejection edit.. About 78 percent of these rejected 
applicants responded properly to the rejection edit or edits and, 
therefore, successfully re-entered the system and received an eligibility 
determination. The possibility exists that many of the 8 percent who are 
currently rejected will re-enter the system, having not- yet responded to 
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EXHIBIT 2.3: SUMMARY OF CURRENT STATUS OF APPLICANTS EVER REJECTED 
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the rejection edit. The 208,278 individuals, or 5 percent of all 
applicants, who have remained rejected for two or more transactions are 
having the most difficulty in negotiating the system. 

There is concern that the low income, needy applicants make up the 
majority of those applicants having the most difficulty negotiating the 
s^^stem and that the rejection edits may be categorically excluding this 
income group from the processing system. In order to address this 
concern, data on the status from previous to current transactions of 
applicants reporting various income and household data on their 1979-1980 
applications was analyzed. The following table shows the income level of 
applicants most likely to become rejected and remain rejected. 

TABLE 2.S: REJECTION RATES BY INCOMEl/ 




A/lncoine equa-ls the sum of adjusted gross income and non-taxable income. 
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As Table 2.5 indicates, applicants reporting negative or zero 
incomes, who constituted about 5 percent of all applicants, became 
rejected and remained rejected at the highest rate. Nearly all zero and 
negative income applicants were rejected under rejection edit F (see 
Sec.tipn 2.2.2 for a detailed examination of this edit), which screens all 
individuals reporting incomes less thant iSl. However, once rejected, 
applicants with negative incomes had considerably more success than zero 
income applicants in re-entering the system. About 17 percent of the 
negative income applicants who ever received a reject iom edit are 
currently rejected, whereas 68 percent of the zero income group who ever 
received a rejection edit are currently rejected. The rejection rates of 
the income groups between SIOOO and $17,500 show little variation. 
Applicants with incomes from $1 to S999 become rejected and remain 
rejected slightly more frequently than applicants with incomes greater 
than $17,500. 

The frequency of rejection of independents and dependents was also 
compared by income level in- Table 2.6. The tab: ^ shows that independent 
applicants were rejected and 'remained rejected at - Mghtly higher rate 
than dependents. However, the. rejection rates of • groups varied 
considerably by Income level. Nearly all zero income applicants,' 
independent and. dependent, were ever rejected. Independents in this 
income group, however, had much more success ifi re-entering the 
processing system following rejection: as of May, 1980, only 24 percent 
of all zero income independents had been rejected fo,' two or more 
transactions, while over ^0 percent of their dependent counterparts had 
remained rejected for this length of time. In addition, independents 
with incomes from SI to S15,0OO were rejected considerably less often 
than dependents in this income range. For example, nearly half of all 
dependents reporting an income between SI and $999 were rejected, whereas 
only one-third of their independent counterparts ever received a 
rejection edit. In sum, the rejection edits are having a greater degree 
of impact on dependents with very low incomes than on the more numerous 
independents in this income group. 
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The following table lists the application charaterl sties. In addition 
to a very low income, which are most likely to trigger a rejection edit: 

TABLE 2.7: CHARACTERISTICS OF APPLICATIONS MOST FREQUENTLY REJECTED 



APPLICATION 
CHARACTERISTICS 


NUMBER OF 
APPLICANTS WITH 
CHARACTERISTIC 


10 

EVER 
REJECTED 


% REMAINED l 
REJECTED TWO OR 
MORE TRANSACTIONS 


MS - DECEASED 


1 7Rf) 




71.6% 


UE - S6,000+ 


9.084 


inn n 


18 • 8 


A6I - BLANK 






73,3 


HS - BLANK 


36 166 




67 ,0 


PHE - BLANK 


31,688 




67 .4 


VEB - $7,000+ 


1,616 


75.2 


34.7 


VEB - $1,001-2,000 


80,102 


63.7 


7.6 


TP - BLANK 


145,610 


61.3 


36.0 


NTI - $12,500+ 


28,486 


57.3 


8.7 
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51.5 


13.1 
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12,698 


50.6 


13.5 


MS - WIDOWED 
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none 



The processing system, as Table 2.7 shows, automatically rejects 
applicants who file as dependents and report that both parents are 
deceased at the time of the original application. As ff.May, 1980, 
of the applicants in this comparative -mall group hLd "e-entered'the 
system and obtained an eligibility determination; ?" i>».-?nt, as the 
table indicates, had been rejected for two or more -.-tions. The 
1979-80 Basic Grant application does not inform the applicant that he or 
she is an. irJopendent if both parents are deceased at the time of the 
first filing, even, though one or both parents may have provided 
assistance, claimed the applicant as a tax exemption, or provided a home 
for the applicant during 1978. The 1980-81 applicant resolved this 
problem: its instructions clearly state that the applicant is 
automatically an indeoendent if both parents are deceased. 
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Table 2.7 also indicates that all applicants reporting unusual 
expenses greater than $6,000— once again, a comparatively small 
numbei'— are automatically rejected. Section 2.2.2 provides a detailed 
examination of the re-jection edit which screens applicants reporting a 
large amount of unusual expenses. 

Applicants leaving key income and household items blank have a very 
high likelihood of becoming rejected and remaining rejected. Also, as 
Table 2.7 indicates, the processing system frequently rejects applicants 
reporting veteran's benefits between 51,000 and S2,000. In fact, 
applicants reporting any amount of veteran's educational benefits were 
rejected at a notably higher rate than applicants not reporting 
benefits: 35 percent of applicants with no benefits were ever rejected, 
whereas the processing system rejected 50 percent of the applicants 
reporting veteran's benefits. 

2.2.2 A dequacy of the Individual Rejection Edits and Cormients 

This subsection assesses the adequacy of the individual rejection 
edits and consents by examining applicant corrections and tha ability of 
applicants to re-enter the processing system following the receipt of a 
rejection edit. More specif i cally, -the following criteria were used to 
analyze applicant response to each edit and cormient; 

• frequency of corrections 

• frequency of verifications 

• magnitude and direction of corrections measured by effective SET 
change and potential payment change 

t ability of applicants to re-enter the system following 
corrections 

• current payment status of applicants who received rejection edits 

Ideally, a rejection edit should correctly identify those with 
missing or inaccurate data. The directions given by the rejection 
comment should be clear and easily understood to insure a complete, 
accurate and prompt response. With this in mind, the following 
assumptions were made in labeling a rejection edit "successful": 
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• A high percentage of applicants will change suspect data items 
in response to a successful edit. Likewise, few will verify 
that the data in question is correct. 

• In order to be worthwhile, an edit should solicit corrections 
that have more than a negligible impact on the applicant's SEI 
and potential grant size. 

• A high percentage of applicants should be able to obtain an 
eligibility determination on the transaction immediately 
following receipt of a successful rejection edit. Thus, the 
majority of applicants who received the successful edit should 
currently be eligible or ineligible. 

Applicants can be rejected for morp than one reason on a single 
transaction. The system, however, notes only one of the reasons, or 
codes, even though all applicable rejection comments are generated on the 
SER. The codes for each rejection edit are given the priority order 
alpha A to 2 then numeric 1 to 17. For example, if an applicant is 
rejected for reasons B and 7, the system notes only code B but the 
student receives comnents for both reasons. In order to gain a more 
accurate understanding of the adequacy of the rejection reasons, the data 
for the analysis in this section is based on the actual reasons for 
rejection— that is, the rejection coirments that were generated— and not 
on the priority rejection code appearing on the applicant file. The 
rejection code, as used in this analysis, provides a convenient way to 
group cormients by rejection reason. 

Some caveats concerning the interpretation of the findings in this, 
section must be stated: 

1) Applicants who received edit comments and who make changes to 
the fields to which the comments apply may be submitting these 
corrections for reasons other than those solicited by the edit 
comnents they received. To the extent that corrections which 
appear to be made in response to the edit comments are made for 
other reasons, the attribution of eligibility index changes to 
the- edits may be overestimated. 

2) Applicants who are rejected do not have an eligibility index. 
In order to assess the impact of the edits, it was necessary to 
compute an eligibility index for applicants on the transaction 
of rejection and compare the difference between that index and 
the index received after they responded to the rejection comment. 
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The computation of the index is confounded by the fact that one 
of the reasons for rejection is missing data. Ideally, the way 
to obtain reliable estimates for missi.j data is to conduct a 
nonrespondent survey. In this case, it would entail selectinq a 
representative sample of rejected applicants with missing data, 
and contacting them to ffnd out true values. Unfortunately 
this study is confined solely to statistical analysis of 
existing data, and it is not within the scope of this contract 
to conduct such a survey. Two alternative methodologies that 
are limited to using existing data were considered for this 
computation. One was to substitute average values from 
applicants who had been rejected, but provided data, for those 
who had missing data. The second was to assume values based on 
other data reported, where possible, and where not possible to 
assume the missing value to be zero. 

Neither cv these alternative procedures is ideal because each 
introduces errors. However, given thf> study goals, a procedure 
was selected that was judged to have the least serious 
1 imitations. 

The limitations of the method of substituting average values are: 

a) The fact that the applicants- (whose average values could be 
used for substitution) provided data made them different 
from those who omitted data. Therefore, from a statistical 
design perspective, these applicants are not equivalent and 

• such a substitution would introduce biases of an unknown 
magnitude and direction . 

b) From an operational perspective, the number of categories 
of applicants who would have to be defined would be 
extremely large, .greatly increasing the time necessary to 
process the computations. 

The limitation of the zero/logically derived substitution 
approach is that it introduces a bias of an unknown 
magnitude, but a known direction. That is, we know that we 
may be using a zero instead of a value. Therefore, this 
procedure introduces a bias which probably understates 
actual values to some unknown degree. 

The approach that was selected was the zero/logically 
derived, substitution approach because the unknown component 
of the error is limited to magnitude and not direction 

• The specific assumptions used for each field can be found 
in Appendix B. 

As mentioned previously, there are 43 rejection . edits for 
nonsupplemental applicants. For this analysis, however, only those edits 
which address critical fields were examined. Table 2.8 provides a list 
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of the number of rejected applicants by rejection reason. The following 
two grouDS summarize the most frequent and least frequent rejection 
reasons: 



F - 



Most Least 
Frequent Frequent 

Reported tax greater c - HS greater than 19 ^ 
than computed tax by / \. 

S300 and 50 percent of A - Reported tax greater than 
computed tax 50 percent AGI 

Portions greater than 120 K - Debts qreat«>r than 
percent or 170 Percent ARI assets - farm 

Zero, negative or less than R - Apolicant's resources 
S5i income decreased by S300 

In addition to the above, a relat-Wely large number of applicants 
received one of the. felling four iPiTect-i-on .edits, given for leaving key 
application fields blank; 7, 8, 16 and 17. Un particular, many 
applicants reported at least one earned-Loeofne portion, but left AGI 
blank. Also, a surprisingly large number of applicants were rejected for 
reporting unusual expenses of greater thcin r. ,v;v 

Table 2.9, which follows Table 2,8, sh. . ; .-^ percentage of al 1 
rejected applicants correcting In response to a rejection comment for 
each of the rejection reasons. The most succesi^ful edits in terms of the 
frequency of solicited corrections were reasons S and T, generated when 
the amount of social security benefits reported^on the Basic Grant 
application does not match the amount recorded on the Social Security 
Administration's computer file. Eighty percent of all applicants 
rejected for reasons S and T corrected NTI on the first subsequent 
transaction, • • 

As would be expected, a high percentage of applicants corrected in 
response tothe four edits associated with missing data. In particular, 
reason 7, which screens applications where both MS and HS are blank, 
solicited a high rate of corrections. Remarkably few changed an 
appropriate field in response to reasons F - zero, negative or less than 
S51 income - and E - UE greater than income or S5,000. This strongly 
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Code/Re.iection Reason Cormients Who Recefvedj/ 



A - Reported tax greater than 259 i 7on 

50% AGI 

B - Combination verify . , 214 134 

10 61,122 ' 

12 17,090 

13 12,264 
128 22,984 
247 16 

297 4 

298 370 

299 334 



C - HS greater than 18 la 

0 - Portions greater than 120% 
or 170% AGI 



506 



190,280 

27 84,828 

41 27,544 

42 ^4,224 
234 33,684 

£ - Unusual expenses greater than 

income or greater than S5,000. 30 ' 109,328 

F - Zero, negative, or lass than 

S51 income . . 173^922 

58 21,170 

98 106,390 

99 18,120 
238 27,370 

300 . 294 

301 96 

302 20 

303 466 ' 

Reported tax greater than . 
computed tax by S300 and 50%. 

of computed tax 47 210 •'^6 

H - Debts greater than assets - home 179 47,442 

I - Debts greater than assets - 

investments 18o 17^30 
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TABLE 2,8: (Continued) 



/n • ^- r. Number 
Code/Rejecti on Reason Corments Who Receivedl/ 



J - Debts greater than assets - 

business 18i 22,880 

K - Debts greater than assets - farm 182 6,244 

L - Unreimbursed tuition greater than 212 77fi 

35!? of income ^'^^^ 



R - Applicant's resources decreased 

by S300 255 

S - SSA match - blank or zeroes - - 



7,044 



39,096 

138 35,570 

287 460 

288 3,066 

T - SSA match - SS benefits 

reported . . 50 

139 . 48,336 

289 238 

290 1,324 



14,422 



U - VA match - blank or zeroes 283 

V - VA reported and less than 

S131 284 34,182 

7 - MS, HS Blank - - 24,110 

70 14,314 

140 9,795 

3 - NTI, AG I and Portions 

all blank . , 27,890 

23 12,580 

43 4,970 

44 1,528 
236 8,400 
307 198 

309 195 

310 15 

311 2 
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TABLE 2.8: (Continued) 



Code/Rejection Reason 



9 - Divorced, widow^sd ; 
unmarried cormients 



16 - AGI blank and portion 
reported 



17 - TP blank and AGI 
greater than zero 



Comments 



24 
33 
35 

106 
148 
270 
308 
313 



268 
271 
273 
275 
277 
306 
315 
316 



251 
269 
279 
280 
281 
282 
312 
314 



Number 
Who Received!/ 



50,766 
0 

31,130 
18,240 
1,280 

0 

0 

116 
0 



146,658 
0 

78,502 
36,876 
8,728 
21.646 
6 

402 
504 



37,164 
4 
48 
18,726 
8,364 
2,968 
5.132 
588 
334 



i/The frequencies are duplicated, given that many applicants 
received more than one rejection reason. 



TABLE 2.9: i]{;g;'-^,«-^,---^^ANr^^ RELEVANT CRITICAL FIELUS U. IMt ... 
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suggests that these two edits, both of which were given to a 
comparatively large number of applicants, are not successfully 
identifying applicants with inaccurate data. Reason G- reported tax 
greater than computed tax by $300 and 50 percent of computed tax - is 
worth examining closely since the highest percentage of rejected 
applicants met this reason, it appears to be effectively identifying 
applicants who misreported TP, with 52 percent correcting this 
application item; to a lesser extent, reason 6 identified applicants who 
apparently misreported AGI, with 32 percent correcting this field. 

Besides changing a critical field, a rejected applicant has the 
option of confirming that the data in question is correct. Table 2.10 
shows, the percentage of applicants, by rejection reason, who verified 
critical fields on the transaction just subsequent to receiving the. 
rejection reason. 



TABLE 2.10: VERIFICATION BY REJECTION REASOM 



REJECT 
REASON 
CODE 


NUMBER 
WHO RECEIVED 
EDIT . 


% WHO 
VERIFIED 


REJECT 
REASON 
CODE 


NUMBER 
WHO RECEIVED 
EDIT 


% WHO 
VERIFIED 


• A 


1,790 


6.14'% 


L 


■ 25,778 


28.325i 


B 


114,184 


5.89 


R 


3,066 


39.52 


C 


506 


13.19 


S 


39,096 


0.02 


D 


190,279 


• 22.94 


T 


50,398 


0 


E 


109,328 


50.32 


U 


14,422 


0 


F 


173,926 


45.79 


V 


34,132 


0 


G 


210,726 


14.67 


7 


24,110 


0 


H 


47,442. 


9.85 . 


8 


27,890 


0 


I 


' 17,130 


15.59 


9 


50,650 


0 


J 


22,880 


45.80 . 


16 • 


146,658 


0 


K 


6,244 


33.28 


17 ' 


37,164 


0.04 



An inverse relation exists between the percent who corrected and the 
percent who verified in response to a rejection edit. A very high 
percentage verified questioned data while relatively few corrected in 
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response to reasons E and F. A high proportion - 74 percent - corrected 
in response to reason H - home debts greater than assets - while 
comparatively few verified. Not surprisingly, almost none verified in 
response to the four edits which request that missing information be 
provided. 

In order to be worthwhile, an edit has to solicit corrections that 
have more than a negligible impact on the applicant's potential award 
amount. Table 2.11, whirh follows, shows the magnitude and direction of 
applicants' corrections to critical fields in response to the rejection 
edits. Before examining this table, refer again to the caveats regarding 
the interpretation of the data outlined earlier in this section. The C 
payment difference in Table 2.11 is based on the potential expected 
disbursement from the time the edit was received to the first subsequent 
transaction. As a point of reference, the expected disbursement is the 
amount of award an applicant is due to receive, taking into account 
enrollment status, cost of education and the SEI. (In the case of the 
rejected .applicant, the expected disbursement is based on the specially 
computed SEI.) 

In general, as Table 2.11 indicates, about half of the corrections 
solicited by the rejection edits did not have an impact on the 
applicant's SEI. The 50 percent who changed their SEI as a result of a ■ 
correction tended to reduce their expected award between .50 and 300 
dollars. However, the corrections in response to the following six edits 
differed notably from the above generalizations: F, H, S, T, 7, and 9. 
Over. 70 percent of the corrections in response to reasons F - zero, 
negative or less than $51 income - and H - home debts greater than 'assets 
- had no impact on the applicants' SEI.' In contrast, fewer than 10 
percent of the corrections to S and T - no social security match - were 
inconsequential in terms of the SEI. Over 80 percent responded to 
reasons S and T with corrections to their disadvantage. The data 
strongly suggest that the two social security match edits are 
consistently identifying misreporting, whereas reason F, given to 
applicants reporting very low income, does not appear to be identifying 
inaccurate information. 
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TAOLE 2.H: Hit WUHIiluI. AND DIRECTION OF COnOECTlOHS IN RESPONSE fU KfJLCrrON fV'- :„ 
TERMS OF EFFECTIVE SEI CHANGE AND AVERAGE POTENTIAL PAYMENT DIFFERENCE " 



Reject Peason i Ji ^ 
Code 



A 


ZWi\ 




7 




67% 


+141 


-J42 


1,680 


B 


70 


•^•817 




.510 






-$392 


170,010 


C 


57 


•^884 


5 








-J 36 2 


302 


D 


27 


^666 


5 


-498 


c / 




-JoZ 


153,236 


p 


48 


^-803 


? 






+ JO / 


-J243 


28,222 


F 


11 


+852 




"•tea 


79 


+6ca 


-$300 


76,522 


6 


53 


+365 


9 




oo 


+150 


t Inn 

-S109 


196,414 


H 


22 


+434 


. 7 




71 
f i 


+30 


t An 


37,044 


I 


29 


+441 


1 ' 


-574 




*a7 


-5/3 


12,368 


J 


25 


+391 ! 7 


-486 






tee 

-155 


8,694 


K 


24 


+575 1 7 








-187 


3,460 


i_ 


44 


+789 1 1 

i 


-336 


55 


+341 




iU , J90 


R 


74. 


+356 


1 


-680 


25 


+251 


•JLOC 


017 

00/ 


. S 


86 


+323 


5 


-341 


9 


+261 


'5l7i* 


11 d44 


T 


82 


+263 


9 


-224 


a 






An cf\a 


U 


38 


+494 


5 




JO 


4>1 44 




c ^ QA 

0 ,ioA 


V 


31 


+264 


25 


-157 




^4 J 




12 «9ao 


7 


9 


+661 


29- 


-fill 




1 nc 

-iOD 


+5128 


29,016 


8 


52 


+922I 


i 


-676 


47 
If 






65,820 


9 


11 


+457 


48 


-1221 


41 


-536 


+S377 


36,844 


16 


38 


+340 


19 




4J 


+60 


-$44 


316,122 


17 


41 


+291 


28 


-451 


31 


-10 


-$7 


74,132 


TOTAL OTHER 

NCM-REJECTION 

EDITS 


.30 


+455 


16 


.447 


54 


+62 


-J17 


272,796 



'MKl^^^'lil^ «nd payment differences are based on the total number of corrections 
nn/n, cpulcal fields. Note that they are based on the number of corrections , 
not on the nuniber of applicants making corrections. 



All the edits, except reason 7 - HS and MS blank - and 9 - number of 
portions and MS inconsistent - solicited corrections resulting in lower 
award potential. It should be noted that it 1s the function of both 
edits to cause the award level to increase. Applicants who meet reason 7 
Mill automatically increase their eligibility by providing the missing 
household size figure. Reason 9 rejects applicants whose marital status 
is inconsistent with the number of portions reported; the data indicate 
that applicants are unnecessarily reporting the income of a second person. 

Corrections to the following rejections resulted in relatively small 
potential award changes: 

H- home debts greater than assets 
V - VEB reported and less than $131 
17 - TP blank and AGI greater than zero 

The ability of applicants to re-enter the system and receive an 
eligibility determination following a correction in response to a 
rejection edit is illustrated in Table 2.12. . As this table indicates, 
applicants responding to reason C - HS greater than 19 - were most 
• . successf L.1 in re-enterfng the system, with only 9 percent remaining 
rejected on the 'f-; Mbsequent transaction. Applicants responding to 
the social security i„ veteran's educational benefits edits (S, T, U and 
V) and to reason 7 - HS and MS blank - were also relatively successful in 
re-entering the system. In fact, applicants responding to. edits that 
required corrections to only one or two fields tended to remain rejected 
less frequently than applicants who responded to edits requiring 
corrections to three or more fields. A comparatively high proportion of 
applicants who corrected in response to reason 9, an edit which demands 
corrections to seven fields, remained rejected. This suggests that 
either the demands of this edit are too severe or that the. instructions 
given by the corresponding comments are unclear and confusing. 

Table 2.13 shows the current eligibility and payment status of 
applicants who have ever been rejected by the rejection reason. This" 
table indicates the degree to which each edit may have impeded applicants 
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TAOLC 2.n: CURRENT ELIGIBILITY mo PmW STATUS OF £V« RCJCCTCO APPLICANTS BY REJECT REASON 
Not t^pected To Dt Pii<j 




J 


51 


U 


20 


20 


49 . 


0.1 


0.1 


49 


22,880 


.K 


45 


9 


16 


20 


55 


0.1 


0 


■ 55 


6.244 


L- 


55 


16 


9 


23 


15 


0.4 


0.1 


45 


25.778 


q 


55 


.11 


17 




45 ' 


0.8 


0.5 


43 


7p044 


S 


45 


12 


. ; 


26 


55 


0.1 


0.1 


55 


39.096 


T 


39 


9 


6 


24 


61 


0.1 


0.1 


61 


50.398 


U 


71 


32 


9 


30 


29 


0.1 


0.1 * 


• 29 


14.422 


V 


44 


: 


20 


19 


56 


0.2 


0.1 


S5 


• 34.182 


7 


70 


es 




29 


30 


0.6 


. 0.1 


30 


24.110 


a 


68 




12 


22 


32 


0.3 


0.1 


?2 


27,390 


9 


87. 


48 


7 


32 


13 


0.6 


0.2 


12 


50.766 


16 


59 


23 


16 


20 


^1 


0.3 


0.1 


41 


146,658 


17 




64 


26 


18 


20 


36 


0.1 


0.1 


36 


37.164 


Total Other 


53 




H 


29 


47 


0.2 


0.1 


47 


I.28G.878 
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from receiving a grant. In general, the data in Table 2.13 corroborates 
the findings from Table 2.12. Applicants rejected for reasons S, T, and 
V, all of which correspond to the social security and veteran's benefits' 
tape matches, were most likely to appear on the recipient file as of May, 
1980. Fewer than forty-five percent of the applicants who received one 
of these three edits wera not expected to be paid. In contrast, 53 
percent of the applicants who received non-rejection edits were not 
expected to be paid. This indicates that rejection reasons S, T, and V 
created no significant roadblocks for rejected applicants. 

On the other hand, reason 9 appears to be a substantial barrier: 37 
percent of those who received this rejection edit were not expected to be 
paid. A high proportion of applicants meeting reason 7 were also not 
expected to be paid. However, the majority in this group were currently 
ineligible, rather than currently rejected as most were who had met 
reason 9. 

The current status of applicants rejected for reason F - zero, 
negative or less than S51 income - is worth examining closely since a 
large percentage of applicants in this group are currently eligible and 
expected to be paid. This either indicates that many applicants have not 
been recorded on the file or that a large number "dropped out" and did 
not submit an' eligible SER to their financial aid administrator. 
2.2 .3 Sumnary and Conclusions 

The following suimiarizes the key findings regarding the impact and 
adequacy of the rejection edits: 

t About 36- percent of all applicants were rejected by the 

processing system for providing missing or apparently inaccurate 
information. As of May, 1<)?", over three-fourths of the 
rejected applicants had re?r>^::i.ded properly to the rejection 
corrments and were given an r.iigibility determination. The 
naqorny of applicants who we^e currently rejected had been 
rejected on two or more transactions. In general, the rejection 
edits form a greater roadblock than the other less restrictive 
edius, with a higher percentage of applicants currently not 
expected to be paid who received rejection edits than those who 
did not. A majority of the corrections in response to the 
rejection edits had no effect on the applicants' SEI. Those 
corrections that did have an impact tended to be to the 
applicants' disadvantage. 
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Applicants reporting negative and zero incomes were rejected at 
a significantly higher rate than the rest of the population. 
Negative income applicants appeared to have little difficulty 
re-entering the system following rejection; however, a large 
percentage of zero income individuals, in particular the 
dependents, remained rejected for two or more transactions. 

iSi fl».if5^Sl]«r?'^"®K^^??P2i""^?.'"^P°'"^^"9 a" ^'ncome less than 
551, fared poorly by all the criteria used in this analysis. 
Very few applicants corrected in response to reason F. 

of those who corrected did not change their award 
potential. Nearly half responded to reason F by confirming that 
the data in question were correct- The data strongly suggest 
that the processing system is creating an unnecessary roadblock 
for very low income applicants, given that it ar^oears that 
reason F is not soliciting award reductions. Cu.versely, the 
data could mean that reason F is a strong deterrent for the 
misreporters. Clearly, further study is needed regarding the 
accuracj' of very low income applicants. If further stud'- -Hws 
that applicants reporting negative, zero or less than ■ icome 
do not misreport more often than other applicants ther. 
consideration should be given to abolishing edit edit f 
Otherwise, the following refinements could be made to the edit 
to make it less restrictive: 

" applicants with negative incomes, given that 

the 1974-75. and 1976-77 IRS studies showed that negative 
income applicants are among the- most, accurate reporters of 
income and taxes; 

- Match low income with household size or number in post-high 
enrollment. For example, reject applicants with one in the 
. household and income less than S500, two in household 
income less than SIOOO, etc. 

In addition to very low income, missing data, taxes inconsistent 
with income and portions inconsistent with AGI were major 
reasons for rejection. About 6 percent of all applicants were 
rejected for leaving a key household or income item blank. Most 
f^^i^nrr un^", ^^^f reported an ^^rirned income portion, but 

left AGI blank. In general, the applicc.nts with missing 
application information had difficulty entering the system, with 
over 60 percent remaining rejected for two or more 
transactions. The data indicates the need for continued 
emphasis in the application instructions of the necessity of 
provKjing complete information. 

In addition to nsjection reason F, reason E - UE greater than 
income or S5,000 - and reason 5 - number of portion inconsistent 
with marital st«-.^;us - performed poorly. Reason E, received by 3 
percent of all applicants, solicited few corrections and had a 
high percentage o^ verifications, suggesting that many 
applicants do in fact report unusual expenses greater than 
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S5,000 even after being questioned by the system. With mors 
higher income applicants than in prior years and because of the 
general impact of inflation (i.e., increased medical and dental 
!^'^®"!f^A;,«''^ follows that many applicants have expenses greater 
than 35,000. It appears, then, that reason E should be 
refined. Making this edit less restrictive by raising the 
S5,000 level would be the most logical refinement. Nearly 90 
percent of the 50,766 applicants who met reason 9 were not on 
the recipient file ap of May, 1980. A comparatively high 
percentage remained rejected on the first transaction after 
receiving edit 9. The texts of the comments corresponding to 
this edit are long; all demand corrections to seven fields. The 
data suggest that the comments' instructions, because of their 
length and relative complexity, are confusing ape'-? "lants and 
impeding many from re-entering the system. Converi^sly, it could 
mean that the applicant misreported and concluded that it was 
not worthwhile to re-enter the system. 

The edits associated with the Social Security and Ve-.^-ran's 
benefit tape matches are the most effective "of the rejection 
edits based on the criteria used in this analysis. A relatively 
high percentage corrected (particularly in response to the 
Social Security edits), almost none verified that the data in 
question were correct, and a very high percentage promptly 
re-entered the system following the receiot of ihe edit. 
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3 

ADEQUACY OF THE PRE-ESTABLISHED CRITERIA 

This Chapter assesses the efficacy of the the pre-established, 
criteria as indicators of misreporting. The Chapter is divided into fou 
major sections, rhe first section (3-1) provides background information 
on the PEC and describes the study design and the measures of 
effectiveness analyzed. The second section (3.2) attempts to determine 
whether selecting applicants for validation because they meet PEC, 
generally identifies applicants likely to misreport better than random 
selection. Section 3.3. examines the relative efficacy of the PEC 
subgroups; the fi nal section (3.4) surmiarizes the findings and presents 
recommendations. 

3.1 Background on the Pre-established Criteria 

Several years ago,, the Bureau of Student Financial Assistance 
developed a set pre-established criteria (PEC) to help identify students 
likely to misreport information, on their BEOG application. Misreporting 
is here defined as the occurrence of post-selection (eligibility) 
corrections which result in SEX changes to the applicant's disadvantage. 
The criteria, based on previous analyses of misreporters , established 
categories of applicants with questionable information on their 
application and/or suspic.JS corrections behavior, and targeted a 
certain number of applicants in each category for validation. 

The PEC have traditionally consisted of three groups: the A group is 
intended to identify applicants. who have been rejected for reporting 
incomplete or inconsistent information, and who, through one or more 
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corrections, have established eligibility; the B group relates to 
applicants who were previously ineligible, and who, through one 
correction have significantly reduced their eligibility index and are now 
eligible; the C group identifies eligible applicants who significantly 
reduce the-ir highest eligibility index with one, continuous, official 
series of transactions. Each PEC group is made up of several subgroups 
or criteria which more finely specify suspect applicant characteristics 
and behavior. In 1979-30, the number of PEC subgroups doubled from the 
number used in the pevious year, eleven new subgroups were added to 
criterion A, three to criterion B, one subgroup to criteria C, and a new 
criterion group D (with 3 subgroups). The D group identifies applicants 
who report having four or more family members in post-high-school 
education and/or applicants whose reported financial data is estimated 
instead of based on actual tax information. The new A subgroups are 
concerned primarily with Veteran and Soci'al Security Benefits and 
crossing B, C, and D criteria with A criteria; the new B and C criteria 
allow for crossing all possible combinations of B, C, and D criteria. 
Appendix D at the end of this report lists the PEC used in 1979.-80 and 
explains their meaning. 

As, of May, 1980, the- cut-off for data for this report, 166,348 
applicants were selected for validation because they met the 
pre-established criteria; 46,014 or approximately 22 percent of the 
applicants validated were chosen randomly. Table 3.1 presents a summary 
of the number of validation applicants selected according to 
pre-established criteria and randomly. 
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TABLE 3.1; SUMMARY OF VALIDATION CEILINGS AND NUMBER OF APPLICANTS 



Criterion 

_6roup 



Final 
C e.i ling 

127,000 
25,000 
17,500 
70,000 
60,000 



# of Applicants 
Selected for Validation 



% of Validation 
Population 



A 



137,618 
21,140 
13,274 
81,266 
46,014 



45.9 
7.1 



B 



C 
0 

Random 



4.4 
27.2 
15.4 



Total - 



299,500 



299,312 



As can be seen in the above table, the majority of students chosen for 
validation were selected because they met one of the A criteria. 

3.1.2 Study Design 

The selection of validation applicants according to two major 
groups - those meeting PEC and those selected randomly, as well as the 
division of those meeting PEC into their aopropriats subgroups, 
facilitates comparisons which can indicate the relative effectiveness of 
each group (and subgroup) in identifying misreporters . In these 
analyses, effectiveness is compared at several levels. First, measures 
of effectivs^npss for the random validation group are compared with the 
same measures of effectiveness for the total group of validation 
applicants meeting. PEC to determine if the PEC criteria in general are 
superior indicators of misreporting. Next, measures of effectiveness for 
the PEC criteria groups (A, B, C, D) are compared by individual groupings 
and by measures of effectiveness for all PEC validation applicants to 
determine the relative efficacy of each group. Finally, the specific 
criterion subgroups within each PEC grouping are compared to each other. 
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3.1.3 Measures of Effectiveness 

Previous studies (including Phase I of tin's study) ha/e attempted to 
assess the efficacy of the PEC primarily through the use of two measures: 

• the percentage of applicants submlttinq post-selection 
corrections; and 

• the magnitude of the SEI changes resulting from these 
corrections. 

The use of the measure "rate of applicants correcting post-selection" is 
based on the assumption that the validation process detects applicants 
who have reported Inaccurate information on their application, and causes 
them to correct the inaccuracies. Corrections, therefore, are synonomous 
with misreoorting. The higher the rate of applicants correcting, the 
greater the number of misreporter.'- identified.!/ The use of the 
.".easure ''magnitude of SEI change resulting from corrections" is a 
necessary romplement to the rate of corrections in judging effectiveness 
since the vi'.us of corrections which result In little or no SEI change is 
questionable at best. This measure is based in part on the assumption 
that the degree of SEI change reflects the magnitude of the discrepancy 
between previously incorrect and' now correct Information. Therefore, it 
follows that the greater the magnitude of SEI change, the more effective 
the criteria in Identifying mi sre porter s .-^ 



i/lt IS assumed that the new post-selection corrections represent 

accurate information. Although study findings support this assumption 
(I.e., post-selection corrections tend to reduce awards), it Is an 
untested assumption because no data are available on the extent to 
which institutions carry out the validation procedures, and the degree 
•to which data are correctly verified. A study of institutionally- 
verified income and taxes is currently being undertaken in cooperation 
with the Internal Revenue Service. This should provide some insight 
into the extent to which institutions have validated these data. 

1/For these analyses, effectiveness of the PEC is determined by the 
magnitude of the SEI change, regardless of the direction of the 

• change. It is assumed that since the correct award of Basic Grants is 
one of the program goals, it Is appropriate for individuals both 
overreporting and underreporting to be corrected. 
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While the rate of applicants correcting post-selection and the 
magnitude of the resulting SEI change are the primary measures of 
effectiveness, we have also used several other measures to address 
effectiveness. These measures provide added depth to the analyses and 
allbw us to focus on the impact of each PEC group in detail. These 



• the percentage of applicants correcting post-selection whose 
corrections do not result in any SEI change (in contrast to the 
gross measure above of applicants correctinr ^ost-selPCti^q 
regardless of whether the correction elicit: ^EI change); 

• the relationship between the reason a student was chosen for 
validation and the critical fields the applicant corrects; 

• the relationship between the percentage of applicants correctinc 
post-selection whose corrections do not result in an SEI change! 
the current eligibility status of applicants and the rate at 
which they appear on the recipient file; and 

• the magnitude of change in a student's expected award resultinq 
from the post-selection corrections. 

It is important to analyze the PEC and random group using the 
above-mentioned measures for the following reasons: 

1. Analysis. of the number of applicants whose corrections do not 
result in SEI change indicates the degree to which the criterion 
groups are refined; that is, the degree to which the criterion 
group Identify only those applicants whose award potential will 
change as a result of validation , and exclude those applicants 
likely to make corrections that do not impact on the student's 
award potential. It is assumed th&t it is most cost-eff*»cti ve 



measures are as follows: 



2. 





Analysis of the relationship between the reason a student 



3. 
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4. Comparisons of the rate at which students appear on the 
recipient file, their current eligibility status, and the 
percentage of applicants submitting corrections not resulting in 
SEI change provide information which allows for the 
determination of whether certain edits are excessively 
students^ ^"^ unnecessary barriers for certain types of 

The following sections analyze applicants selected for validation in 
terms of the above measures. Section 3.2 compares randomly-selected 
applicants with the total group of applicants selected because of PEC. 
Section 3.3 attempts to analyze the relative effectiveness of each PEC 
criterion by making within PEC group comparisons. 

^•2 Oyerall Effectiveness of the PEC as Coiro.ared to Random Selection 

This section assesses the overall effectiveness of the PEC relative 
to random selection as an indicator of misreporting. To be considered 
effective at this broad level, the PEC must, at a minimum, elicit a 
higher rate of applicants correcting and a greater resulting SEI change 
than the random group. These indicators provide some idea as to whether 

the concept of targeting specific groups of students for validation is 
appropriate. 

The following' table (Table 3.2) shows the percentage of PEC and ■ 
random applicants making post-selecti on corrections, the resulting SEI 
change, and data on several other factors indicative of effectiveness. 
As can be seen in the table, approximately ten percent more PEC 
applicants than random applicants corrected any field post-selection. 
Although both applicant groups increased their SEI, the SEI change for 
the PEC applicants averaged almost 90 points more than for their randomly 
sellected counterparts. Furthermore, the ratio of PEC applicants raising 
their SEI to lowering their SEI is about twice that of the random group. 
The difference between the average amount SEI was raised. and lowered was 
quite small- for both groups, however, (-f-g for the PEC group, and -9 for 
randomly selected applicants). The percentage of applicants appearing on 
the recipient file is also similar i^or both groups (approximately SU) , 
So, too, is the current eligibility status of the applicants. 
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TABLE 3,2: OVERALL COMPARISON OF EFFECTIVENESS OF RANDOM SELECTION 
VERSUS SELECTION ACCORDING TO PEC 





P.EC_ADp1i cants 


Randomly : ' 
Sejected Aopli cants 


% of Applicants Correcting Any 
Field Post-Selection 


38% 




Average S£I Change 


128 


37 


Composite Change Inde;<i/ 


4,364 


1,073 


% of Applicants Correcting Any 
Field With No StI Change 


33 • 8% 


64.1% 


% of Applicants Correcting 
Critical Fields Post-Selection 


325S 


23% 


Ratio of Applicants Raising 
SEI to Lowering SEI 


3:1 


1.7:1 


Difference Between Average 
Amount Raised and Amount Lowered 


9 




% of Applicants Correcting . 
Critical Fields With No 
SEI Change 


20« 


29^0 


% of Applicants on 
Recipient File 


56.2% 


58.2^ 


Current Eligibility Status of all 
Applicants 
% Eligible 
n inengiDie 
% Rejected 


99.4% 
.C3% 
.47% 


99. 6S 
.05?^ 
•32/9 


Average S Difference Between 
Selection and Current 
Pajonent Transaction!/ ' 


-$20.2 


-33.1 




^'s an overall score of effectiveness 
averagi S^I^'hSnSlr"^ °^ applicants correctin^by thiir 

^''^l^fpienf^le"'*'""*^^ ^^'^'''^^ applicants who are not on the 
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Approximately 99.5 percent of all validation applicants-both PEC and 
random-are currently eligible. Between five and eight-tenths of one 
percent are currently ineligible, while less than one-half of. a percent 
is rejected. Reflecting the significant difference in the percentage of 
applicants increasing their SEI and the magnitude by which the average 
SEI increased, PEC validation applicants have a much greater dollar 
difference between the average expected award at selection and that at 
payment than do randomly-selected applicants. The expected award 
declines by approximately twenty dollars between selection and the 
current transaction for PEC validation applicants, but by only about 
three dollars for randomly-selected validation applicants. 

It can be concluded from the above data that, their totality, the 
pre-established criteria are more effective in identifying misreporters 
than the random selection group. This indicates that the basic concept 
of selective action strategies is sound. Applicants selected by this or 
a similar method will tend to have a high.er percentage of applicants 
correcting and greater resulting SEI change than applicants selected for 
no specific reason. The, data further indicate that, as would be 
expected, positive SEI change translates to smaller average expected 
awards. This means that applicants selected for validation based on the 
existing pre-established criteria will generally have a more significant 
reduction in potential award than randomly-selected students, resulting 
in fairly substantial savings. One additional piece of data that rrray 
also indicate that the, PEC are better than random selection in 
identifying misreporters is that the percentage of applicants correcting 
whose correction do not result in SEI change is lower for the PEC group 
than for the random group, despite the fact. that the overall correction 
rate is higher for applicants meeting ?£C. Ona possible reason for this 
is that the PEC are indeed more sensitive- to factors that result in SEI 
change. In sum, these analyses suggest that the development of criteria 
identifying misreporters is a concept that should be pursued. 
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3.3 Relative Effectiveness of PEC Subgroups 

This section analyzes the relative effectiveness of the 
pre-established criteria. Although the PEC can generally be considered 
more effective than random selection in identifying students reporting 
inaccurate information, it is necessary to examine each PEC criterion 
separately to determine relative effectiveness when compared with other 
criteria and the random selection process. Previous studies have 
indicated that there is substantial variation in the degree of 
effectiveness within the various PEC subgroups. Certain PEC subgroups 
consistently score high v<hen judged on effectiveness, while others score 
equal to or lower than the random group. A third group tends to be 
moderately effective. As with the previous analyses, effectiveness of 
the criteria is assessed primarily in terms of the rate of applicants 
correcting post-selection and the resulting average SEI change. 

Table 3.3, on the following page, summarizes applicant corrections 
behavior and the resulting SEI change by PEC criteria and for the random 
applicant group. The table shows the number of applicants selected for 
validation for each criterion subgroup and group (A, B, C and D); the 
percentage of applicants correcting post-selection within each group; the 
average SEI change, the composite change index; and the percentage of 
.applicants whose corrections do not result in SEI change. It should be 
noted that the table is based on corrections made to any field, not just 
to critical fields. Corrections counted include such things as address 
changes, and systems generated transactions, which do not result in SEI 
change. Therefore, the percentage of applicants whose corrections do not 
result in SEI change will be higher and the magnitude of the average SEI 
change will be lower than that found on Table 3.6, which analyzes 
correction made only to critical fields. Both tables are presented in 
this analyses to show that a significant number of the post-correction 
are made to other than critical fields. 

3-3.1 Rate of Applicants Makinc) Post-Selection Corrections 
Tcble 3.3 Clearly show;; that there is substantial variation in the 
rate of applicants submitting post- select ion corrections to any field 
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CRITERION 
GROUP 



NUMBER . 
S£L£CnD 



X OP APPLICANTS 
CORRECTING 9CST. 
SECTION 



AVERAGE COMPOSE 
SEI CHANGE 
CHANGE2/IH0EX3/ 



% OF ApMLICA.ns 
CORRECTING mO 
SEI CHANGE 



P£C AdoI leant 

•PEC GROUP A 

A*l 

A-3 
A-i 
A-5 
A«»6 

A-r 
A-e 

A-o 

A-:o 

A-li 
A-12 
A-I3 
A-I4 
A-I5 
A-16 
A-17 
A-IS 
A-19 
A- 20 
A.51' 



GrouO 



Group 



Ranaoni 



3-2 
3-3 
3-4 
8-5 
5-0 



C-1 
C*2 
CO 

0 

D-2 



U6,348 

137,613 
2.596 
6S,300 
36,580 
13,130 
9,576 
44,948 
7,930 
S,986 
2,67i 
21,822 
12,276 
2,720 
46 
7,120 
1,426 
436 
7,006 
773 
3,746 
5,862 
63,342 

21.140 
636 
20,472 
44 
134 
410 
4,040 

12,274 
1,630 
. 11,674 
415 
2,542 

31,266 
13,406 
76,362 
11,002 

46,014 



38 

38 
34 
28 
37 
2c 

39 

42 

25 

34 

34 

49 

31 

30 

42 

45 

52 

29 

.20 

29 

27 

h 

40 

29 

22 

29 

36 ■ 

22 

39 

38 

29 
34 
22 
36 
35 

42 
51 
<2 
56 

29 



128 

138 
90 
«v « 

175 
53 
162 
233 

a 

103 
74 
122 
137 
138 
101 
113 
103 

- 26 
125 

62 
131 
77 
*117 

98 

- 22 
102 
233 

57 
92 
102 



73 
117 

96 
116 

106 
75 
108 



4,864 
5,244 

3,060 
2,420 
6, 475 

9.786 
735 
2,502 
2,516 
6,462 
6,987 
4,140 
4,242 
5,527 
5,356 
•1,044 
2,7SC 
1.79S 
2.527 
2,211 
4,680 

2,542 
- 725 
2,958 
8, 288 
1,054 
2,627 
3,375 

3,219 
2,482 
3,276 
2,456 
4,060 

4,452 
3,825 
4,536 
4,256 

1,073 



33. c 

25.3 
44. £ 

22.0 
45.2 
77.9 
45.5 
23.5 
24^9 
26,3 
24,8 
12,7 
11,2 
34,3 
40,0 
is, 9 
15 ,9 

10 ,'9 
31,: 
25 .4 

35 ^4 
40 ,'3 



^ * 

36 

12.5 
32 .2 
20 !2 
■ 20.9 

34.5 
22.9 
36.5 
25 ' 
24^4 

29.5 
23. w 
28.7 
17.0 

55. c 



l/Corrections perwin to all fields, no: on]y critical fields. 

l2^?oi^^in?^h!l''cS^^^ ^^'^'!^ ^-^ ^"'^^"^ cnanoe for applicants rais^n'^ 

?n1luo°;^r?^osf^^^^^^ Hunger or applicants cor^ec:?^L"^ 

l/'nj Comooslte Change Inaex is an overall scco of effect i van e«:? n«-i«-r 
aiu.nplving :he percent 0' aoplicants correctini % iZ'l^ZTl' ?n%]rl£pchan9e. 
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when compared by PEC group and subgroup. PEC group D had proportionately 
more applicants correcting post -selection than any other group (42%). 
PEC A fonowed with 38 percent of its applicants correcting. PEC groups 
C and B tied for x^m lowest percentage of applicants correcting 
post-selection (29'^:). The rate of corrections for PEC C and D group 
applicants equalled, but did not exceed, that of the random selection 
group. 

The variation in the percentages of applicants correcting 
post-selection is more evident when PEC subgroups are examined. The 
table indicates that some PEC subgroups are extremely effective in terms 
of a high rate of applicants correcting, while others are less effective 
than random selection. The following subgroups have the highest and 
lowest rates of applicants correcting: 

TABLE 3.4: PEC WITH HIGHEST CORRECTION RATE 



PEC With the Highest Correction Rate 


PEC With Lowest Correction Rate 


Applicant 


Magnitude of 


Applicant 


Magnitude of 


Subgroup 


SEI Change 


Subgroup 


SEI Chanqe 


D-3 ' 


56% 


B-I4 


22% 


A-I5 


52% 


• A-I9 


27% 


D-1 


51% 


C-2 


28% ! 


A-11 


51% 


A-4 


29% 


A- 10 




A-16 


29% 


A- 13. 


49% 


A-18 


29% 


A- 20 


43% 


B-2 


29?; 


A-6 


42% 


A- 12 


30« 


0-2 


42% 


A- 17 


Z0% 


A-21 


40% 


A 


Z0% 



3.3.2 Average Effective Scl Changes 

In contrast to the findings of prior analysss of correction rates, 
groups A, B, C and D have similar average SEI changes. Group A has 
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an average change of 138 points, while groups C and D have average 
changes of ill and 106 points, respectively. PEC group B trails, but 
insignificantly, with an average 98 point SEI change. 

Further analyses of SEI change shows that there is considerable 
difference in the magnitude of SEI changes resulting from post-selection 
corrections when the changes are compared within PEC subgroups. The 
following table shows the PEC subgroups with the greatest and least SEI 
change. 

TABLE 3.5: PEC WITH GREATEST AND LEAST SEI CHANGE 



PEC With the Least SEI Change 



PEC With the Greatest SEI Change 



Applicant 
_ Subgroup 


Magnitud'e of 
SEI Change 


A-7 


+21 


B-1 


-22 


A-16 


-36 . 


A-4 


53 


B-4 


57 


A-18 


• 62 



Applicant 
SubgrouD 


Magnitude of 
SEI Change 


A- 6 


233 


B-3 


233 


A- 3 


175 


A-2 


155 


A- 5 


162 


A-12 


138 


A-U 


137 


A- 10 


132 



.♦Random group SEI change: 43 points. 



As can be seen in the table, with the exception of criterion B~3 the 
PEC subgroups with the greatest SEI change are clustered in the A group. 
PEC subgroups and A-6 (income and taxes paid inconsistent), A-3 (unusual 
expenses inconsistent with income or greater than $5,000), A-2 (portions 
earned and AGI inconsistent), and A-5 (unreimbursed tuition greater than 
35% of income) had post-selection correction which resulted in large 
Dositive SEI changes, overall SEI change for applicants correcting 
from PEC subgroup B-3 was very large; however, the change was in the 



3.12 



82 



negative direction. Apolicants in PEC subgroups A-1 (missing 'iformation 
on household size or marital status), A-10 ( no SS match and applicant 
verifies). A- 11 (no SS match and applicant changes), A-12 (VA blank or 
zero and applicant verifies) and A-19 (combination of an A criterion with 
C-2) also had a hiqh degree of SEI changes. The average change for these 
groups was 35 points. 

The groups with the lowest magnitudri of change were fairly evenly 
distribut... oetween the A, B, and C groups. Both A-7 (financial data 
missing) and B-1 (large SEI change) had SEI change levels 6 points below 
that of the random group. Applicants in subgropus C-1 (large SEI change) 
and A-16 (combination of an A criterion and B-1) also had corrections 
which resulted in less SEI change than the randomly-selected groups. 

3.3.3 Composite SEI Change 

Another measure of the PEC's effectiveness in identifying potential 
misreporters is the composite SEI change index. This index is computed 
by multiplying the percentage of applicants correcting post-selection by 
the resulting average SEI change. It is based on the assumption that 
given the current system model which has one treatment for all 
applicants, the criteria determined most effective should be those that 
result in the highest overall yield to- the Department of Education, both 
in terms of the percentage of • applicants correcting and the magnitude of 
SEI changes. Using the composite change as an indicator, the following 
subgroups can be said to be most and least effective. 

Therefore, the higher the number of applicants whbse correction result in 
an SEI change, the more effective the criteria. 

3.3.4 Percent of Applicants Correcting Post-Select ion with no SEI 
Change " ~" 

One final measure of effectiveness analyzed in Table 3.3 is the 
percentage of applicants correcting post-selection, whose corrections do 
not result in SEI change. Analyses of this measure provide further 
evidence of effectiveness by looking at the consistency by which the PEC 
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TABLH 3,6: MOST EFFECTIVE PEC AS DETERMINEu BY COMPOSITE C '.N6£ INDEXl/ 



PEC 
Subgroup 

A-6 
B-3 
A- 11 
A- 3 
A- 10 
A- 5 



% Of Applicants 

Correcting 
Post Selection 2 / 

42 
36 
5"". 
37 
49 
39 



Average 
SEI Change 

233 
233 
137 
1/5 
132 
162 



Composite 
Change Index 

9,786 
8,388 
6,987 
6,475 
6,468 
6,318 



♦Composite score for random group: 1073 



i/Subgroups are ranked in order of decreasing effectiveness when judged 
on comoosite ^core. ^ 

^/Represent; 



:s individuals correcting any field post-selection 
regardless of whether the correction resulted in an SEI 



change. 



TABLE 3.7: LEAST EFFECTIVE PEC AS DETERMINED BY COMPOSITE CHANGE 
INDEXl./ 



% of Applicants 

^ f-'' Correcting Average Composite 

SubarcjuE Post Selection 1 / SEI Change Change Index 

?i 33 -22 - 726 

^5 21 - 735 

A-16 29 .36. .l/oil 

29 . . 53 1537 

A-18 29 . 62 1.798 

^Composite score for random group: 1073 



l/Subgroups are ranked in order of decreasing effectiveness when judged 
on composite score. ' 

2/Represents individuals correcting any field post-selection, 
regardless of whether the correction resulted in an SEI change. 
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distinguishes between serious inisreporters whose corrections will result 
in significant SEI change and other applicants. Idenlly, the PEC should 
be very discriminating; to make validation most worthwhile only those 
students who have reported very inaccurate information on their 
application, and whose corrections will result in large SEI change should 
be selected for validation. 

Data on Table 3.3 show that over one third of all validation 
applicants correcting post-selection submit corrections that do not 
f-esuU in an SEI change. The data also show that the rate of PEC 
applicants correcting without SEI change is only slightly less than the 
rate of randomly selected applicants correcting without change (33.8^ and 
39.4^ respectively). Within the PEC subgroups, however, there is wide 
variation in the percentage of applicants correcting who have no SEI 
change. For example, over three quarters- of all applicants correcting in 
PEC subgroup A-4, submitted corrections which had no impact on SEI, while 
only 7 percent of the applicants correcting in. the B-1 subgvoup had no 
SEI change. The table shows that the subgroups least consistent in 
identifying individuals whose corrections affect their award potential 
are (in order of greatest inconsistency): A-4 (77%), A-12 (55%), A-S 
(45%), A-3 (45%), A-13 (40%). The most consistent PEC subgroups are: 
B-1 {7%), A-15 (11%), A-11 (11%), B-3 A-10 (13:"^), A-15 (16%), D-3 

(17%) and A-14 (19%). 

TABLE 3.8: MOST AND LEAST CONSISTENT PEC GR0UP5 



PEC Subgroups Most Consistent 
in Identifying Students 
With SEI Change 


PEC Subgroups Le!ist Consistent 

in Identifying Students 
_ . With SEI Chanqe 


B-1 


A-15 


A-4 


A-16 


D-3 


A-12 


A-11 


A-14 


A-5 


B-3 




A-3 


A- 10 




A-13 



3.15 



It should be noted that there are sev'^ral reasons why the rate ■ 
corrections not resulting in SEI change seems to be high. First, Tc 
3.3 is based on corrections made to any field, and as such, inc'.-des 
corrections (such as address changes and systems generated trans. ...s) 
that do not affect a students SEI. A certain percentage of non-SU 
change corrections can, therefore, be attributed to these factors. 
However, our analyses suggest address changes and the like make up only a 
small percentage of the total corrections. Table 3.4 on the following 
page, shows that almost 85 percent of the post-selection corrections are 
made to critical fields and therefore have the possibility of causing SEI 
change. The second reason that the correction without SEI change rate 
may be high is that the- applicants correcting have negative SEIs, whicha 
re set to zero by he eligibility formuala, that changes, even fairly 
major ones, do not raise their SEI above zero. Finally, the rate may be 
high because the corrections made by the applicant are so minor that they 
don't alter the student's SEI. regardless of the initial SEI. 
3.3.5 Change to Critical Fields 

Table 3.9 is included in this report because, it can be argued that 
analyses of corrections made to critical fields tend to provide more 
precise information, on the number and impact of corrections th^.. data on 
corrections made to any field. Precision is increased because the 
corrections rate and - /rage 'jE I change scores are not diluted by 
including in the population those acplicarits making corrections that by 
definition cannot result in SEI change. 

Despite the differences in the way the correct-:ons rate and average 
SEI change were computed in Tables 3.3 and 3.9, the PEC subgroups that 
are most and least effective when judged by the composite irjex (the 
score that takes into account both the rate of applicants correcting and 
the average SEI change) are similar. The subgroups considered most 
effective when analyzed according to critical field coorections are 
subgroups A-3, A-5, A-6, A-11, and B-3; the subgroups which continue to 
be least effective are A-4, A-7, A-16, A-15, and B-1. 
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Table 3.9 is included in this report for two other reasons. First, 

it allows one to probe deeper and examine not only the average SEI 

change, but also the number of applicants missing and lowering thier 

SEI, Second, it shows that a 1 arge :nuniber of applicants correcting 

without SHI change are making their^lcorrections to critical fields. This 

has significant implication regarding the reliability of some PEC 

subgroups. The high rate of applicants correcting with no SEI change 

cannot be excused as resulting merely from address changes and similar 
factors. 

Analysis of the percentage of applicants and their SEI show that 
there is a substantial difference in the ratios of those increasing and 
decreasing their award ootential by PEC group and subgroup. Overall the 
number of PEC applicants correcting to their disadvantage outweighs the 
number correcting to their advantage by almost three to one. Random 
applicants also submit corrections that result in more applicants 
incr 3ing their SEI than decreasing it; however, the ratio of random 
applicants raising lowering their SEI is significantly lower than that 
for PEC applicants. Comparison of the four PEC groups shows that 
applicants in PEC group A tend to. on the average, increase their, Scl 
mo'-9 often than applicants in the other PEC groups. The ratios of 
applicants increasing their SEI to decreasing it. is as follows fryr each 
PEC group: A (3:1); B (1.5:1); C.(2.o-:l); D (2.5:1). 

The variation in the ratios of applicants correcting to their 
disadvantage anJ advantage within PEC subgroups are greatest in groups A 
and B. In group A, the proportion of applicants increasing their SEI up 
to those decreasing it ranges from 1.2:1 for subgroup A-7 to 5.5:1 for 
subgroup A-12. In group B the ratios range from 1.5:1 to one subgroup 
where no applicants decreased their SLI (subgroup B-3). As mentioned 
previo-isly. the value of t.ie numerical averages presented for PEC 
subgroups B-l, B-3, B-4 and B-S is somewhat questionable because of the 
small number of subjects in the study population. 

Analysis of the diffarence between the magnitude of SEI change of the 
applicants correcting to their disadvantage and these of applicants 
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correcting to their advantage fin contrast to the number of students 
raising and lowering their SEI), indicates that the average S£I change of 
applicants decreasing their award potential is only slightly higher than 
that of PEC applicants decreasing their potential. The average amount by 
which random group applicants increase their award potential, on the 
other hand, is slightly more than the amount of those decreasing their 
potential award value. 

As shown in the analyses of other effectiveness measures, there is 
much variation in the behavior of applicants within the PEC subgroups. 
This variation is evident in- the analysis of the average difference 
between the amount SEI's were raised and lowered, and the direction of 
the net change. For example, in 18 of the total 34 PEC subgroups, the 
magnitude of chsivge in one direction outweighs the magnitude of SEi 
change in the opposite direction by more than 100 points. However, in 12 
group:> the net difference is less than 50 points. Similarly, the net 
dif*!:renc8 between the average amount the SEI was raised and lowered, was 
positive for 29 of the PEC subgroups, but negative for the 
randomly-selected group and other PEC subgroups. 

Finally, as mentioned previously, this table shows that most of the 
corrections not resulting in S£I change are being made to critical 
fields. (In addition, when corrections to critical fields are analyzed, 
PEC subgroups continue to have exceptionally high. rates of applicants 
with no SEI change.) This indicates that current criteria are not 
adequately refined to identify misreporters , and that other methods of 
selecfng applicants should be studied. 

3«3.6 Relationship of Corrections Behavior of PEC Vali dation 
Applicants to "Suspqct" FieTgr" ' ' — ~~ 

The majority of the A ana all of the D PEC criteria are based on the 
assumotion that potential misreporters can be identified solely because 
of the absence of certain information on their application or the 
presence of inconsistent, illogical or other questionable application 
datj, Tt is assumed that either the illogical or inconsistent data are 
•ir.joi'. w:, and when \'2l ■dated will be corrected, or that other related 
data an? inaccurate 5n-> will also be corrected as a result of validation. 
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TA8LE 3.10: PERCENTAGES OF VALIDATION APPLICANTS WITH POST-SELPniN 

CORRECTIONS TO SUSPECT FIELDS BY PEC GROUPA/ 




Shaded 
group. 



areas denote critical fields considered "suspect" for each PEC 
Bold face numerals signify areas of greatest change. 



i/Social security benefits are included 



as part of Nontaxable Income. 
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In assessing the ovej^all effectiveness of the criterion groups, it is 
beneficial to analyze this relationship between the reason a student was 
chosen for validation (the PEC criteria met) and the critical fields 
changed by the student post-selection. Although one cannot assume that 
there will always be a direct relationship between PEC subgroup 
applicants and suspect fields changed, one can assume that the stronger 
the relationship, the more sensitive the subgroup in identifying 
misreporters . 

Table 3.10 on the preceding page shows that there is some correlation, 
albeit a snail one, between the reason for selection and the suspect 
fields corrected for most A and D PEC subgroups. This suggests that 
ther-e is a large random element in misreporting. The PEC subgroups with 
the highest rate of corrections to suspect fiolds, hence the strongest 
correlation, are A-2, A-3, A-6, A-7, A* 10, A^U, A~13, A- 14, A-15, and 
D-20. It is interesting to note that these subgroups are among the ones 
considered most effect. ve when evaluated in terms of the percentage of 
applicants correcting and the magnitude of the resulting SEI change. 
These groups are also among the most consistent in eliciting corrections 
that result in SEI changes. PEC subgroups A-1, A-4, and A-9 have the . 
most tenuous relationship between the reason for selection and 
corrections to corresponding suspect fields. In most cases these 
subgrouDS have also performed relatively poorly when evaluated on the 
other efficacy measures. 

More important than the data presented above. Table 3.10 clearly shows 
that AGI and Taxes Paid are the fields corrected most often regardless of 
the PEC criteria meet. Portions Earned and Nontaxable Income are also 
corrected at a much highe- rate than other fields, but at a rate slightly 
lower than AGI and Tax^s Paid. It is difficult to determine why these 
fields are corrected at such a high rate; however, one reason may be that 
tax and income information is relatively easy to verify and therefore may 
be subjected most rigorously to validation. 

Analysis of PEC subgroup A-5 yielded some part.cularly interesting 
findings. This group has a relatively high percentage of applicants 
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correcting and a relatively high degree of SEI change, indicating that it 
is a fairly effective criterion. Yet, it does not show a high percentage 
of applicants correcting the unreimbursed tuition field post-selection. 
Instead, a relatively large percent of the corrections made by A-5 
applicants are made to income-related fields, specifically AGI and Taxes 
Paid. This suggests that the ratio of the percent of unreimbursed 
tuition to income currently used for the criteria is fairly refined in 
identifying individuals who under report their income. 

3-3.7 I^Ate °f Ap plicants Appearing on the Recipient File and Their 
Current tligibi I ity Status • — 

We have determined the rate at which PEC and randomly-selected 
applicants do not appear on the recipient file and the current 
eligibility status of these applicants. These factors, although not 
clear indicators of .effectiveness, allow us to generate some hypotheses 
on how PEC and randomly-selected validation applicants negotiate the 
processing system edits and validation. 

Table 3.11, on the following page, showsthe percentage of validation 
applicants not appearing on the recipient field as of May, 1980 and the 
percentage of these applicants who are currently eligible. The table 
indicates that there are only slight differences in the percentage of 
applicants expected to be paid and the percent currently eligible, for 
the total PEC applicant and random applicant groups. Except for PEC 
group C, there is also very little difference in these scores when 
compared across PEC groups. For example, the percentage of applicants 
expected to be paid averages 43.8 percent for the A group. 42.0 percent 
for the 8 group, and 43.4 percent for the D group. The percentage of 
applicants on the recipient file is also similar for these threa groups 
(A = 94.1%; B = 94.8%; D = 93.5%). On the other hand, PEC group C has 
quite different averages. Both the total percent of applicants not 
expected, to be paid and the rate of currently eligible applicants within 
this category are much lower than for the other PEC groups. 
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TABLE 3. a 



PERCENTAGE OF VALIDATION APPLICANTS NOT EXPECTED TO BE PAID AND 
THEIR CURRENT ELIGIBILITY STATUS 



Criterion 
Group 



% Not 
Expected 
To Be Paid 



% Not 
Expected 
To Be Paid 
But Cur- 
rently 
_ Eligible 



Criterion 
GrouD 



% Not 
Expected 
To Be Paid 



% Not 
Expected 
To Be Paic 
But Cur- 
rently 
El iqible 



JPEC 

Applicants 43.8 
A 44.7 
A-1 59.2 
A-2 44.2 

A-3 42.6 

A-4 49.7 

A-5 42.1 

A-6 45.2 

A-7 40.6 

A-8 41.2 

A-9 38.9 

A- 10 . 44.6 

A-ll 44.2 

A-12 64.0 

A-13 65 . 7 

A-14 39.0 

A-15 36.6 

A-16 39.7 

A-17 di.2 

A-18 34.4 

A-19 37.4 

A-20 41.7 

A-2I 44.5 

B 42.0 

B-1 . 40.2 



94.2 

93.9 

95.2 

93.4 

93.1 

96.1 

93.2 

91.1 

93.6 

94.7 

93.0 

94.4 

94.3 

96.3 

93.8 

93.44 

94.6 

91.9 

93.9 

97.8 

97.9 

95.0 

93.2 

94.8 

89.1 



B-2 


42.0 


95.0 


B-3 


50.0 


90.9 


B-4 


34.3 


100.0 


B-5 


40.5 


92. S 


B-6 


43.0 


92.3 


C 


35.5 


97.4 


C-1 


33.6 


96.7 


C-2 


35. a 


97.5 


C-3 


27.4 


100.0 


C-4 


36.2 


95.9 


D 


43.4 


93.5 


D-1 


40.1 


94.7 


D-2 


43.6 


93.4 


D-3 


40.2 


94.2 


Random 


41.3 


96.5 
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There 1s more variation 1n scores within PEC subgroups, although even 
within these groups the variation is not great. T'. subgroups with the 
highest percentage of applicants not on the recipient file are: A-1 
(59 .256). A-4 (49. 7«), A-5 (45.2%). A.12 (64. 0?:), A-13 (66. 75^) and 8-3 
iSO.0%). Only PEC subgroup C-3 has a percentage of applicants not on the 
recipient file that 1s substantially lower than other groups (27.4%). 
The subgroups with the greatest percentage of not expected to be paid 
currently eligible applicants are a-4 (96.1%), A-12 (96. 8«), A-IS 
{97.8%), A-19 (97.9%), 8-4 (100%), C-1 (96-7%), C-2 {97.5%) and C-3 
(100%). PEC subgroups A-6, A-10, A-U, A-13, A- 14, A- 15, A-18, B-3, and 
C-4 also have the greatest dollar difference. These groups also 
correspond to the groups with the highest SEI changes. In all but 
subgroup A-13, the difference results in savings to OSFA. It appears 
that applicants in A-13 tend to underestimate their financial need and 
that validation assists these applicants in obtaining greater awards. 

. PEC subgroups A-1, A-4, A-7, A-9, A-16, and B-l have the lowest 
dollar difference between expected payment at selection and at the 
current transaction. These subgroups had an average dollar savings of 
only •*9,069 per subg>oup. This is not suprising, however, since these 
subgroups have performed poorly on almost .every other effectiveness 
measure. 

Analysis of these figures with the figures on the percentage of 
applicants correcting whose corrections do not result in any SEI change 
Table 3.11 suggests that there nay be some correlation between these 
numbers. For example, the subgroups with the highest percentage of 
applicants not on the recipient file (A-1, A-4, A-12, A-13, and 8-3) tend 
to have either very high or very low percentage of applicants correcting 
without resulting SEI change. The following table sunmarizes these 
relationships. It can be Hypothesized that high not expected to be paid 
scores combined with a high percentage of applicants correcting' without 
SEI change indicates unnecessiiry barriers placed in the way of applicants 
1n the groups having these scores. The edits may also be restrictive and 
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TABLE 3.12: HYPOTHESIZED RELATIONSHIP BETWEEN APPLIANTS NOT ON THE 

RECIPIENf FILE AND APPLICANTS CORRECTING WITHOUT SEI CHANGE 



S» Expected ^be Paidj/ ^h^^tlgj^g^^Lj?""^^"^ 
A-1 59.2 44.6 
A-4 49.7 77.9 
A-12 64.0 54. g 

A-13 66.7 40;o 

B-3 50.0 12.5 



i/A 
P 

i/A 



perclnt."°* expected to be paid score for PEC applicants is 43.8 



ppr!2ni?f correcting with no SEI change for 

PEC applicants is 33.8 percent (ses Table 3.3). 

may be eliciting a higrt rate of unnecessary corrections. Similarly, a 
high percentage of not expected to be paid, appl icants. combined with a 
low percentage of applicants correcting with no SEI change, could 
indicate subgroups effectiveness in identifying actual misreporters 
(B-3). The hinh percentage of applicants correcting whose corrections 
results in an average increase SEI suggests that the criterion is fairly 
refined in identifying individuals who are serious misreporters. The 
fact that a relatively small percentage of these a^>j>li cants appear on the 
recipient file could indicate that either (1) the misreporting was 
detected during validation and the resulting corrections rendered the 
student ineligible or (2) that the threat of validation frightened many 
misreporters out of the system. 

Examination of the percentage of currently eligible applicants who 
are not expected to be paid lends credence to these hypotheses. As can 
be seen in Table 3.1 , both very high and very low percentages of 
currently eligible applicants not on the recipient file correlate with 
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TABLE 3.13: ESTIMATED DOLLAR SAVINGS FOR CURRENTLY ELIGIBLE VALIDATION 
APPLICANTS MOT ON THE RECIPIENT FILE 



3erween Seisciion 
1 Cjr-ent "?'2nsacrion2/ 



MOD ;^c2n:s 



32V '!nQ53/ 



^I'Z APDl-icants 
•jrouo A- ; 

3 



--19 



3- . 

3- 2 
5- 3 
3- * 
5- : 
3- 3 



"oral Va)iaat?on 



-20.2 
-20.9 

-2:*: 

- 2.3 



-2: .2 

-22.7 
-20.2 
-24.3 



5o, 550 
57,315 
1. -62 
2S.3£3 
:4.522 





3,505 


. 0.5 


:.3HS 




?,2Q1 


.3^.4 




-15.-: 


:,57£ 


-36.7 






1,555 


.22!f 




• 3.0 


159 


.15,0 


2.^13 


-32.0 


"255 


-2^ .3 




•13.3 




-19.5 


25 , 252 




s 


. 7 ,i 


2^ 


•11.5 


3.170 


•76 .2 


20 


* 3 ,T 


as 




15*: 


-15 . " 




-22 .5 




•32.3 


520 


-22. i 


-.067 


-<w.o 




•35 .1 


1,Z29 



32.991 
5.351 

^.in 



37.25C 



SI ,33/, 325 
l,30S.iir 
15,320 
56 7. '306 
315.14? 
1:2.503 

ir,:i2 

'•^,132 
Z.-^IC 
51.137. 
11.315 

25.310 

1.101 

32.730 
ig lex 

572 
51 .661 

32 .'33 9 

514 , 656 

35.950 
550 

5C.34: 
• 1.52u 
^23 



:C3.300 
17 ,386 
51. ICC 



132,31:" 
535, 5/i 
i01,335 



i.-:=;:?c 



*-34v:?gs -s sue to 'ower ;ur-snt. jxDectsc savmen: :nar s: sei-ction. 

'|r=°S.:SnSr«^^^^^ =lus synods in..c3re tne 

signl'lcantiy n^gner aesenamg on tns numoer of stuoents wnc n.ve- ^nr-- in •-- 
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very high percentages of the total applicants not expected to be paid. 
Examination of subgroups of particular interest above, (A-1, A-4, A-12, 
A-13, B-3), shows that the high percentage of currently eligible 
applicants in groups A-1, A-4, and A-12 supports the hypotheses that 
applicants in these groups are being unnecessarily hindered by the 
current edit and validation systems. Conversely, the low percentage of 
currently eligible nonrecipient applicants found in D-3 suggests that 
these applicants may be legitimately excluded from receiving awards. 
3.3.8 Expected Dollar Savings 

The final measure for analysis in the study of the effectiveness of 
the PEC is estimated dollar savings. As can.be seen in Table 3.13, 
significantly greater savings can be expected by selecting validation 
students based on PEC criterion instead of randomly. Students selected 
for validation by PEC but not currently on the recipient file, have SEI 
changes which may result in substantially greater savings for the PEC 
than the random group . 

Within the PEC groups, PEC groups A and D had the greatest average 
dollar difference between the amount of expected payment at selection and 
the amount on the most current transaction, thus indicating the two most 
promising large groups for selection. As would be expected, these groups 
are also the ones with the greatest SEI changes (see Section 3.2). 
3.4 Sunfmary of Findings 

Our analyses of the various measures of effectiveness for the pre- 
established criteria indicate that there is wide variation in the 
effectiveness of the group and subgroups in identifying students likely 
to misreport. The analyses indicate that in general, however, the 
concept of selecting students for validation according to specific 
characteristics that tend to be indicative of misreporting, . i .e, , 
inconsistent taxes and income information, is sound. This is evidenced by 
the fact that the overall average scores for PEC applicants are higher 
than for random applicants. Analyses of the four PEC group A, B, C, and 
D suggest that only groups A and D consistently identify a relatively 
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high percentage of misreporters. Groups A and D have overall composite 
change indices that are greater than for the indices of B and C groups 
PEC group A (4864); B (2842); C (3219); and 0 (4452). In contrast to the 
B and C groups which identify students for validation based on the 
students current and previous questionable transactional history, group A 
and D are based on current application data such as veteran's status, 
inconsistencies on income and tax information. This has implications for 
information stored on all applicants file suggests that current data on a 
students application form may be sufficient to predict mi sreporting. 

There is also much variation in the efficacy of the PEC subgroup. 
When judged on a composite score taking into account both post-selection 
corrections rate and ;iverage SEI change, criteria A-3, A-5, A-6, A-10, 
A-U, A-14, A-15 and B-3 can be considered most effective and criteria 
A-1, A-4, A-7, A-9, A-16, A-18, and B-4 least effective. 

Seven of the above mentioned effectiveness criteria are particularly 
interesting because they had high scores on the gross measures of 
effectiveness (percent of applicants correcting and average. SEI change) 
and also tended to be quite refined in identifying applicants whose 
corrections wre not inconsequential, but -instead resulted in SEI change. 
In addition, applicants in these groups tended to have a lower rate of 
currently eligible applicants considered, not expected to be paid, 
possibly indicating that these students are in fact, misreporting and are 
either being detected at validation- and forced to make corrections 
(possibly ones excluding them from the system) or else are being 
frightened out of the system by the prospect of validation. These seven 
pre-established criteria and their scores in the efficacy measures are 
seen in Table 3.14. 

Similarly, there are at least two subgroups that neither elect large 
net composite SEI change nor are particularly refined. The subgroups are 
A-1 and A-4 (see Table 3.14). 
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Anniir,nf IV^f^^^ Uorrection Hot Expected 

JP "'^^"' Uorrectiiig SEI Change With No to be Paid but 

^""^^'^'^P Post-Selection Cliange Index SEI Change Currently Eligible 



A^5 (unreimbursed tuition M . 162 6310 45 S;^ 
greater than M of . ' " 

income) 



A«6 (A61 and Taxes Paid m . 233 9786 23.5^ 911^ 
Inconsistent) 

A-SO (Ho SSA match and m m 6468 12 7^ mm 

applicant verifies) ^^'^^ 

A-U (No SSA matcli and 513$ 137 5937 u ov 04 n 

applicant changes) ^^'^^ 

^-1'' if 0 $131. m ■ 133 5537 m 93 4^ 

applicant verifies) '^^ 

A'15(VA 0 $131, IO3 5356 I5.9X 93 4X 
applicant changes) 

B-3. (combination B-1 and 36X . 233 8388 12.5^ 90.0% 



TABLE 3.14: THE LEAST EFFECTIVE AND REFINED PliE-ESTABLISHED CRITERIA 



, Average Composite Uorrection J! Not Expected 

5PP''':^"t Uorrecting SEI Cliange With No to be Paid but 

^"Pg^Q"P . Post-Selection Change Index . SEI Change Currently Eligible 

A-l(Househo1d Si2e or 34 90 3060 44.6!^ 95 2^ 
Marital Status Missing) 

A-4 (now Income Applicants] 29 53 1537 77.9^ 96.11^ 



VALIDATION AND THE INTERACTION OF VALIDATION AND EDITS 

Each ear financial aid administrators at institutions of higher 
education across the country validate approximately 10 percent of the 
Basic Grant applicant popoulation. As of May 1980, 212,362 individuals 
had been selected for validation and had presumably documented their 
financial status to officials at the school they plan to attend. 
ADproximately 7S percent of the applicants were chosen for validation 
because, they meet one or more of the pre-established criteria; the other 
22 percent were selected randomly.i'' 

This chapter analyzes the impact of validation!/ on applicants 
post-selsctipn/eligibility corrections behavior and SEI change, and the 
relative impact of the' processing system edits and validation on the 
correct award of Basic Grants. The first part of this Chapter (Sections 
4.1-4.3) examines the pre- and post-selection/eligibility corrections 
history of validation and nonvalidation applicants to determine whether 
the applicant groups are- comparable, and to ascertain which 
pre-established criteria are most affected by validation. The primary ' 
analytic measures used in this section are (I) the rate of pre- and post- 
selection/eligibility corrections, (2) the resulting average SEI change, 

1/See page for discussion of meaning of random selection. 

1/More accurately, this chapter analyzes the impact of being selected 
for validation since there is no way of knowing whether validation 
actually took place. 
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(3) the percentage of applicants correcting post-selection without 
changing their SEI, and (4) the percentage of applicants expected and not 
expected to be paid and their current eligibility status. The second 
part of this chapter (Section 4.4) examines the interaction of the edits 
and validation to determine which process is better in assuring that! 
applicants report correct application information. The validity of 
critical data elements at various points in the edits and validation 
process is the primary measure for analysis. 

^•1 Impact of Validation-Comparison of P re-and-Pn!;t Selection/ 
Eligibility Change " — — 

The following table (Table 4.1) shows the pre- and post-selection/ 
eligibility corrections behavior and resulting average SEI change for 
validation and nonvalidation applicants. The table is divided into all 
PEC applicants, PEC group and subgroup applicants, and those randomly 
selected or not meeting the PEC. 

Analysis of all validation and nonvalidation PEC group A applicants 
shows that the rate of applicants correcting pre-selection/eligibi lity 
and the resulting SEI. change is almost identical for both groups. 
Seventy-four percent of all PEC validation applicants corrected 
pre-selection and had corrections which resulted in an average 99 point 
SEI change; seventy-three percent of the nonvalidation PEC group A 
applicants corrected pre-eligibility with a resulting 98 point average 
SEI change. 

There is, however, very substantial difference in the post-selection/ 
eligibility behavior of these two applicant groups. Almost five times 
more PEC group A validation applicants made post-selection corrections 
than nonvalidation group A applicants (32% of validation applicants 



i/Before preceding further, it should be mentioned that although 

^Sfl^Jn?" tl^ PEC validation applicants with all nonvalidation PEC 
applicants is the ogical first measure for analysis, this comparison 

mLtiL''plr'5rn,m/p "".^"^^TS ^^^P^^'" ^' Honval i dati On app 1 i cants 
meeting PEC groups B, C. and D were not flagged by the central 

^rSJnf S"" ^'"^ therefore not correctly identified in the data base and 
cannot be compared. 
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and 6% of the nonvalidation applicants corrected post-selection/ 
eligibility). The resulting average SEI change is also significantly 
different for both groups. The average SEI change for group A validation 
applicants is 155 points; the average change for group nonvalidation 
applicants is 38 points. Thus, the average post selection SEI change for 
validation applicants is over three times greater than for nonvalidation 
applicants. 

Comparison of the corrections rate and SEI change for validation and 
nonvalidation PEC A applicants strongly indicates that validation is 
having an impact on applicants' post-selection/eligibility corrections 
behavior and resulting SEI change. Almost 20 percent more validation 
applicants than nonvalidation applicants corrected post-selection/ 
eligibility, and their corrections resulted in an average 120 point 
greater SEI change. 

Comparison of randomly selected validation applicants and 
nonvalidation applicants not meeting PEC showed that these group behaved 
differently than the groups mentioned above. Randomly selected 
validation applicants corrected pre-selection almost five time more often 
than nonvalidation applicants {25% and 6% respectively). In addition, 
although the magnitude of the SEI change waS' not that different for the • 
two groups (44 points for validation applicants; 26 points for 
nonvalidation applicants), the direction of SEI change was dissimilar. 
Ramdomly selected validation applicants generally increased their SEIs 
pre-selection while on nonvalidation applicants decreased their SEIs. 

Analyses of the rate of validation and nonvalidation applicants 
correcting pre-selection/eligibility by specific criterion subgroups 
shows that there is very little difference in the rate at which one 
validation subgroup corrects and the rate at which its companion 
nonvalidation subgroup corrects^ The only exceptions to this are 
subgroups A-9 and A-12. Proportionately, one third more validation 
applicants than nonvalidation applicants corrected pre-selection in 
subgroup A-9, while fifty percent fewer validation applicants corrected 
pre-selection in subgroup A-12. 
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There is greater variation in the average SEI change for the vali- 
dation and nonvalidation subgroups. Although, all but three subgroups 
have net SEI changes in the same direction (the exceptions are subgroups 
A-4, A-7, and A-12) the difference in the magnitude of the changes varies 
significantly. For example, there is a 270 point difference between the 
average SEI change for validation and nonvalidation A-9 subgroup 
applicants, but only a 14 point difference in the SEI change for A-11 
applicants. 

Both validation and nonvalidation applicants in subgroups A-2, A-3, 
A-5, A-6, A-7, A-10, A-11, A-14 made pre-selection/eligibility 
corrections which resulted in net positive SEI change. Validation arid 
nonvalidation applicants in subgroups A-1 and A-9, however, made changes 
which resulted in a net decrease in SEI. Validation applicants in 
subgroups A-4, A-8, and A-12 decreased their SEIs pre-selection, while 
their nonvalidation counterparts initially increased their eligibility 
index. 

Analyses of post-selection corrections and average SEI changes for 
validation and nonvalidation applicants in PEC A subgroup showed that 
with the exception of four subgroups (A-4, A-8, A-11, -12), between five 
and six times more validation applicants corrected post-selection than 
nonvalidation applicants. The. SEI change resulting from these 
corrections differs within the subgroups. The difference between the 
average validation SEI change was greater than 150 points for subgroups 
A-2 and A-6, between 100 and 150 points for A-1, A-3, A-5, A-10 and A-12, 
and less than 100 points for subgroups A-4, A-S and A-9. The SEI change 
for validation applicants in subgroup A-4 was only slightly above the 
change for comparable nonvalidation applicants, while the SEI change for 
validation applicants in subgroup A-7 and A-9 was less than that for 
similar nonvalidation applicants. 

4.2: Impact of Validation - Percent of Applican ts Correct inq 
Host-Selection with No S£l Change 

Analyses of miscellaneous data elements brought to light one element 
which seems to indicate that validation helps applicants make appropriate 
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decisions on when it is necessary to correct their application 
information. Table 4.2 shows that although over four times as many 
validation as nonval idation PEC applicants corrected (38% to 9%, 
resi:,ctively), the rate of validation PEC applicants with no resulting 
SEI change is only about half that of the nonvalidation group (33.8% to 
64.1%). The figures for the random group are similar. This suggests 
that the realtively small percentage of nonvalidation applicants who 
correct are making inconsequential, and most likely random corrections. 
The figures for the validation group, on the other hand, indicates that 
validation definitely helps refine the types of corrections made. It may 
be that contact with financial and administrators provides the added 
information students need to correct properly. 

In^Pact of Validation- - A pplicants on (not on) the R ecipient File 
and Their Current Flirfih ility ^^^a^•■.c -'^ — 

One could hypothesize that validation would have significant impact 
on the rate at which applicants appear on the recipient file. The 
following table (Table 4.3) addresses this issue; however, it provides no 
evidence that validation either facilitates or impedes an applicants 
process through the system. The table indicates that as of May 1980, 
appro:<imat8ly 57. percent of all validation applicants appeared on the 
recipient file. Over 99, percent of these applicants were considered 
currently eligible for an award; less than, one-half of one percent were 
considered to have received payment erroneously because they are novy 
classified as rejected or uneligible. 

In contrast, almost 49 percent of all nonvalidation applicants were 
on the recipient file as of May, 1980. Again, over 99 percent of these 
applicants and -currently eligible, and only slightly over two- tenths of a 
percent were paid in error. 

It is difficult to use the data in Table 4.3, to ascertain how 
validation assists or hinders applicants in negotiating the processing 
system through to payment. Although a higher percentage of validation 
applicants are on the recipient file than nonvalidation applicants, this 
does not necessarily mean that validation facilitates payment. One can 
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TABLE 4.2: POST-SELECTIOMS CORRECTIOMS,!/ AVERAGE SEI CHANGE AND PFRcfnt hp 



App Meant 
Group 


% Correcting 
Post-Selection/ 
Eligibility 


Average 
SEI Change 


% Correcting With 
No SEI Change 


Val i dati nn 








Meeting PEC 
Random 


38% 
29 


128 
37 


33.3% 
39.4 


Nonvalidation 








Meeting PEC 
Not Meeting PEC 


9 
7 


24 
29 


64.1 • 
59.7 



i/Corrections refer to corrections made to any field. 

conclude from this data,, however, that validation applicants seldom stay 
currently ineligible or rejected while over 55 percent of the 
nonvalidation applicants are current ineligible or rejected. 

In summary, validation is having a significant impact on most PEC subgroups 
studied, by causing more applicants to correct than overwise would have 
corrected. In addition, most validation applicants have corrections which 
result in greater SEI change than nonvalidation applicants-. Finally, 
validation seems to help students correct more selectively (that is, 'make 
corrections that affect their SEI, not random corrections that do not affect 
their potential award), but. yet does not significantly impede their progress 
through the processing and award system. 



4.7 

107 

ERIC 



ma 4.3: SUHHilRy OF EXPECTED PAYMENT AND CUflflENT ELIGIBIUTV STATUS OF 
VALIDATION AND NONVALIDATION APPLICANTS 



Validation 
Applicants 

Meeting PEC 

Random 
TOTAL 



Expected .to_bejajd NoUniec^^ 

Tnhi n'^'u ^ ['''''^^^ ^"''^^"tly Currently Curfently 

Jotal^^Eligib^ Ineligible Rejected Total Eligible ineligible Rejected 



%M 99.43 
58.17 99.61 
56.60 99.47 



.08 
.05 
.08 



0.47 ^im 94.22 
0.32 41.82 96.51 
0.44 43.39 94.70 



3.80 
2.09 
3.44 



1.97 
1.38 
1.85 



Nonvalidation 
Applicants 

Meeting PEC 64.57 99.22 
' Not Meeting PEC 46.09 99.86 
TOTAL 40.63 99.74 



•Ofi 0.68 35.42 97.90 0.63 
•02 O.Il 53.90 37.46 44.45 
•03 0.21 51.40 43.19 40.30 



1.37 
18.07 
16.49 
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The Interaction of the Impact of Validation an d 
rue Processing iivstem bdits ~"" 

In the analj/ses thus far, an attempt has been made to separate the 
impact of validation and the impact of the processing system edits. In 
Chapter 2, the analysis centered on the immediate response of all 
applicants subsequent to receiving a processing system edit. In 
examining the impact of validation, the analysis has focused on the 
post-selection corrections behavior of validation applicants. The 
validation process and the edits, however, do not affect two separate 
populations. About 84 percent of all validation applicants received a 
processing system edit. Therefore, for a substantial portion of the 
validation population, the edits are working in tandem with the 
validation process to elicit complete and accurate data. 

Table 2.2, in Chapter 2, showed that the majority of validation 
applicants' pre-selection corrections were in response to the processing 
system edits and that these corrections tended to-be to the applicants' 
disadvantage. In particular, PEC applicants' made frequent corrections to 
their disadvantage in response to the edits. In sum. Table 2.2 suggests 
.that the edits- had a greater impact than the validation process on the 
corrections behavior of validation applicants. 

In order to establish more fully the impact of the edits relative to 
the impact of validation, the pre-selection and post-selection 
corrections behavior of PEC Group A validation applicants' was compared. 
It should be noted that all PEC Group A applicants receive a rejection 
edit prior to selection. The rejection reason is identical to the 
selection criterion. In Table 4.4 which follows, each PEC A 
sub-criterion has been matched with its corresponding rejection reason in 
order to compare the pre-.selection influence of the rejection edit with 
the post-selection influence of institution validation. 

Table 4.4 clearly shows that, regardless of the rejection/selection 
reason, the edits are having a much more significant impact than 
validation on validation applicants' corrections. For example, over tO 
percent of the validation applicants who received the Social Security 
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match edits corrected NTI prior to selection, whereas, once selected, 
less than 2 percent changed NTI. The data from this table strongly 
suggest that the edits are soliciting the proper corrections prior to 
selection. 

Table 4.5 demonstrates further the interaction of effects of 
validation and edits. In this table, only the 81,702 validation 
applicants who were on the recipient file as of May, 1980 and who 
received an edit prior to selection to one or more of the seven critical 
fields which financial aid administrators are required to validate are 
examined. Values for these validation fields were compared at various 
points in time to determine whether the edits or validation were 
responsible for the valid data item. The following four categories were 
used to make this determination: 

^ " Applicants whose value at oa'/ment does not eq ual value at 
selection and does not equal value at ecTTtT " ' — ~~ 

In this type of case, applicants did not make the necessary 
changes in response to the edit; therefore, the edit was either 
not effective or partially effective and validation was 
necessary. 

Applicants whose valu e at payment does not equ al the value at 
selection but equals the value at edif ! " 

In this type of case, the edit caused the applicant to make 
unnecessary or invalid corrections and validation aided the 
applicant in reporting accurate data (i.e., the valuewas 
changed to what it was at the receipt of the edit). 

Applicants whose value at payment is equal to t he value at 
selection and not equal to the value at eaTt? ~ 

In this case, the edit caused the applicant to make the 
necessary correction before the applicant's selection for 
validation, rendering validation unnecessary. 

D — Applicants whose value at payment is equal to the v alue at 
selection and equal to the value at edrt ! ^ " — 

Both the edit and validation in this case were unnecessary. 

To facilitate the interpretation of this table, the following 
surtmarizes the four categories: 

A - validation necessary, edits ineffective 

B validation necessary to counteract errors caused by edits 



8 - 



C - 
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UlllilfSi'llllDIIIG rniiEitiA 



Kejcclinn Reason/ 
Selecllwi Criterion 

fori Ions Crcalerlluii im 
or IM AGI 

llnusinl fxpenSRS Grejler Itian 
iMct* or Greater lliaii |5,00n 

ho, Nuijatlve or less Ilian 
151 Inrume 

llepurlcil Jax Greater Than Gondii leil 
laK by \M and m of Cwipiiled I« 

OiireMursed Tuition Greater Than 
M of jiicnue 

A|)pllcaiil'5 licsources 
Decreased liy \M 

SSA Hatch ■ lllank or Zeroes 

SSA Hatch - SS Benefits Reported 

VA Halcii • Blank or Zeroes 

VAReporlcd and less TlianJI3|. ■ 

Harllaj Status, iiwiseliold Sl2e Blaiu 

AG' •■'liiik anil Portion Repurted 

h. M Blank and AGI ■ 
Grt-ater tlian Zero 



Prc-SeUflliin Corrections In iicsponse lo filll 



Reject 
[dit 



1/ 



Percent 



1/ 



317< 

13.3 

I2.S 

30,0 

27.1 

B.I 
iiS.3 
61.0 
13.8 
25.Z 

60.0 
51.4 



Average 
Potent lal^' 



Correcting I' ayiAcnl Change 



■J 70 



217 



55 



24 



222 



• 130 

- 96 

• S9 

- 61 
^ 155 

• 5 

- 10 



UnsnllclledPost-Selccllun Corrections 



l/Peicenlajc cori-ecting relevant field on transaction Jnsl sobsei|(icnl t» receiving edit. 
Z/Cjiciilated fnw receipt of edit to first snhseiinenl transaction. 
3/Peru'ntage corrcctlna relevant field helween selection transaction and cnrrent transaction, 
VCalculaled Iron selection transaction l« corrcnt transaction. 



Criteria 
A-2 
A-3 

A.6 
A-5 
/1.9 

A-IO, A-ll 
fl-lO, A-ll 
A-i2, A-n 
A-11, A-15 
A-l 
A-fl 



3/ 

Percent-' 
Correcting 



ll.OK 

0.5 

2.0 

19.0 

2.0 

0.1 
1.0 

1.8 
0.7 
2.5 
3.7 
9.2 

7.9 



Avcraje 



1/ 



roteiiliaH 
Payiiicnl Ciiarige 

•)I55 

• 130 

■ 75 
•Z21 

• 35 

0 . 

■ I9« 

■ 190 

- Ml 

■ 75 

- 93 

- no 

- 1(1 
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C - edits effective, validation unnecessary 
D - neither edits nor validation necessary 

As table 4.5 indicates, validation applicants misreported AGI most 
often. For 29 percent of the applicants, neither the edits nor 
validation were necessary to elicit a valid AGI figure. By comparison, 
the edits and validation were unnecessary in nearly 75 percent of the 
cases to insure valid PHE. 

The edits, regardless of the field, were more responsible for valid 
data items than the validation process. For over 50 percent of the 
applicants, the edits were responsible for valid AGI by soliciting large 
corrections to the applicants' disadvantage. This corroborates the 
findings from Chapter 2 which suggested that the numerous and restrictive 
edits which address AGI were working effectively. In certain cases, the 
edits caused applicants to make erroneous corrections. Three percent of 
the applicants, for example, im'properly changed HS to their advantage in 
response to the edits. 

In over 80 percent of the cases, the items to be validated are 
accurate before validation takes place. This is due either to the 
original validity of the fields or the effectiveness of the edits. 
Validation was needed only 4.4 percent of the time to solicit valid PHE. 
It was needed most often in regard to AGI, with 17.7 percent of the 
applicants having an inaccurate AGI at the time of validation. In 
several cases in Group A where validation was ultimately necessary to 
elicit a valid field, the edits were partially effective; that is, 
applicants made corrections in the proper direction in response to the 
edits. 
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ERROR-PRONE MODELLING STUDY 



5.1 Purpose and Scope 

OSFA is interested in the identification of applicants who ara likely 
to be misreporting on their 8E0G applications. This study focuses on the 
development of an error-prone model (EPM) related to the validation 
System. 

Any investigation of the characteristics of applicants whose forms 
are likely to contain errors must provide an operational definition of 
errors which include a means of detecting them. This report addresses 
errors that can be detected through the validation process ' employed by 
institutions in the 1979-1980 academic year. 

An overview of this study is found in Chapter 5, along with a 
description of the process through which the model presented in this 
report was arrived at. Chapter 6 presents the model in its final form, 
while Chapter 7 describes each of the. groups intp which applicants were 
classified as a result of the analysis. Chapter 3 presents a summary of 
the EPM study and compares the error-prone model developed here with the 
existing PEC. . 

5.2 Key Study Objectives and questions 

Whereas the previous chapter addressed the impact of the existing 
methods for controlling errors (the edits and validation by 
pre-established criteria) the broad objectives of the error-prone 
modeling process undertaken in this investigation are: 

• to investigate the characteristics of students most likely to 
misreport information on their application, and 
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• to provide a means of identifying such applicants and of 
minimizing the impact of their misreporting 

The specific objectives of the error-prone. modeling component include 
the following: 

•. to provide a means of estimating the likelihood that an 
applicant is misreporting 

• to provide a method of selecting applicants for validation 

• to provide a model that is sensitive to future changes in the 
BEOG program, (such as different restrictions in eligibility 
determination) 

• to provide information for management imorovements soecificallv 
related to: ■ . j 

the validation system 

the edit system 

the original application forms 

reduction of drop-outs among students entitled to funds, and 
a global quality assurance strategy 

In more informal terms, the model presented in this report attempts 
to address the comment, "different corrective measures are needed for 
different kinds of applicants^" by providing a useful operational 
definition of the term "kinds of applicants." . 

5.3 Study Population and Samples 

The study population for this report is all regular Basic Grant 
applicants who had an eligible transaction processed by the end of August 
1979, This date was chosen after examination of the 1973-1979 data as 
one assuring that most applicants re-entering the system after selection 
by that date had been processed at the time the study was conducted. 
Thus the following sets of applicants were excluded from the study: 

• . Applicants whose first eligible transaction was processed after 

August 31, 1979 

• Applicants who were never eligible 

• Applicants who filed supplemental forms 

The samples used in this study are described in Chapter 7, but two 
issues concerning the population and its relationship to the sample must 
be discussed here. First of all, the samples through which the 
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oiode'l was developed consisted entirely of applicants selected for 
validation through the "random" process. This process was not entirely 
r-andom, since early filers with multiple eligible transactions had a 
greater chance of being selected. 'Whatever biases are represented in the 
random validation sample will be reflected in this study. 

The second issue refers to independent applicants affected by the 
change in the eligibility index formula which became effective May 9, 
1979. Unlike the study presented in Part I of this report, Part II 
retained .those applicants. Changes were made to operational definitions 
to account for the systems-generated transaction these applicants 
received, but two factors must be taken into account. 

First, independents who filed prior to May 9 and were ineligible 
under the old formula, but became eligible under the new, could not have 
been selected for random validation if . they only filed one transaction. 
Therefore, although they should. have been in the sample, they were not. 

Second, the degree to which these applicants' behavior was affected 
by the different SEI formula and the subsequent change is unknown. For 
example, the extent to which the change in the SEI computation affected 
aoplicants willingness to re-enter the processing system, proceed with 
validation or- change certain fields. Is not known. Although these 
applicants are included in the sample, possible biases due to their 
presence are taken into account in interpretation of results. 

Four samples, drawn from the population of 1979-1980 eligible 
applicants, were used in this investigation. Each of the first three 
samples consisted initially of 20,000 applicants, approximately 14,400 
dependents and 5,600 independents. Only 10,000 dependents were used in 
each of the statistical procedures, but cases were weighted accordingly. 
Each of the four samples used in the analysis is briefly described below. 
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The workin^q lagole consisted of randomly selected validation 
applicants through the random process whose selection transaction had a 
process date prior to August 31, 1979. The cutoff date was necessary, 
since only the first update of the recipient file was available at the 
time the analysis was conducted. It also enabled us to distinguish as 
well as possible between applicants who failed to re-enter the system and 
applicants whose payment transaction was simply In process. An 
examination of a subsample of the 1978-1979 data revealed that of random 
validation recipients with no post-selection transaction, 73. 85^ of those 
selected by August 31 appeared in the first update of the recioient file, 
while on]y 20% of those selected after that data appeared in the first 
update. Most of the statistical analyses were conducted using this 
sample. 

The reolication sample was selected in exactly the same way as the 
working sample. The majority of the .random validation applicants were 
selected for either the working or the replication sample, but there was 
no overlao between the two. This sample was .used to verify that results 
obtained from the working sample. ware not due to chance. 

The nonvalidation sample was drawn by simulating the random ' 
validation process. ■ The sampling procedure selected 20,000 eligible 
transactions processed by August 31, 1979 belonging to regular 
applicants. Once again, the number of dependent applicants was reduced 
to 10,000. This procedure resulted in not merely a sample of applicants, 
also a simulated selection transaction for each applicant. The 
simulation program was executed without knowledge of the 
systems-generated transaction produced to inform independent applicants 
of the change in the SEI formula. As a result, some systems-generated 
transactions were chosen as the simulated selection transaction. Those 
cases (approximately 200) were eliminated from the sample, resulting in a 
possible minor underrepresentation of certain kinds of independent 
applicants in the sample. This sample was used to distinguish the 
effects of validation from those of edits or spontaneous changes. 
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The PEC sample consisted of 15,000 applicants selected for validation 
on account of having met a, PEC on a transaction processed prior to August 
31, 1.979, This sample was used to compare the effectiveness of the PEC 
to that of the error-prone model. 

5.4 Research Methodology . 

A brief description of the research methodology will be presented 
here. For a more detailed description see Appendices 0 and E. 

An analytical .method known as sequential search analysis or automatic 
Interaction detection (AID) was used to classify applicants into groups 
which differ as much as possible in a dependent variable relating to 
their response to validation. The groups are defined in terms of a set 
of predictor variables. Only variables available at the time of 
selection were used as predictor variables, including SER fields, 
variables related to corrections history, and variables obtained through 
he algebraic manipulation of two or more SER fields. AID first splits 
the sample into two groups which are as different as possible, and 
continues this process for each resulting subgroup. 

The computer program used in the' analyses was THAID, a version of AID- 
designed for nominal scale dependent variables. THAID analyses were 
conducted using the working sample. The effects of each split were 
cross-validated through the replication sample. Those splits that could 
not be replicated were not used in defining the resultng groups. 
Finally, applicants who met the definitions of the groups, but were not 
selected for validation were examined to ascertain the reasons for 
failure to re-enter the system. 

5.5 Dependent Variable 

Several dependent variables were considered for this investigation. 
Quantitative variables (selection to parent discrepancies in SEI, 
scheduled award or expected disbursement, or their absolute values) 
ignored students who did not re-enter the system after selection for 
validation. For this reason a decision was made to use a nominal scale 
dependent variable and a program such as THAID which could handle such 
variables. I- 
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Prior to defining the dependent variable one must define the 
criterion transaction. This Is the transaction which ;vin be assumed to 
be the most accurate and against which data on other transactions will be 
comoared to estimate the effects of validation, or at least of selection 
for validation. The criterion transaction for this study is defined as 
follows : 

• If the aoplicant was paid at or after selection, the latest 
paym'ent transaction 1s considered the criterion transaction. 

• If the applicant was not paid at or after selection and the 
latest transaction is different from the selection transaction; 
then the latest transaction is the criterion. 

' If the applicant was not paid on the selection transaction and 
filed no post-selection transaction, then the criterion Is 
missing. 

t Otherwise (i.e., the applicant was not paid at selection and the 
latest transaction 1s rejected) the applicant was excluded from 
uhe study. 

The previously mentioned systems-generated trat taction, which 
notified a small proportion of the applicants of he change In the SEI 
formula, was ignored in all calculations. If an ,plicant was paid on 
this transaction, the payn:ent was assigned for' -tical purposes to 
the latest pre-May 9 transaction. 

Given this definition of the criterion transaction, the dependent 
variable can be defined. This variable will be referred to as TYPE and 
the term "Type of applicant" will be operationally defined as the value 
of the variable TYPE for the given applicant.- The four types of 
applicants are: 

Iffiel: Exact reporters . Applicants for whom the SEI of the 
criterion transaction is less than 50 points above or below the 
sti of the selection transaction. 

T,yi?g 2; Over-cl aimers. Applicants for whom the SEI of th^ 
cnWion transaction is at least 50 points higher than the' SEI 
of the selection transaction. 

Type. 3: Under-Cl aimers . Applicants for whom the SEI of the 
criterion transaction is at least 50 points lower than the SEI 
of the selection transaction. 

Typg. f*' ''hissing . Applicants with a missing criterion 
transaction. 
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Exhibit 5.1 presents the definition of applicant types in flow-chart 
form. 

The use of expected disbursement, scheduled award or some other 
standard such as "made corrections to some critical field" were 
considered as possible alternatives to a nominal scalp dependent 
variable. The first two alternatives were rejected, since 
institution-related information could have been unavailable if the 
criterion transaction were not a payment one. The use of critical field 
corrections was rejected since it would not discriminate between 
significant and insignificant changes or between overr-cl aimers and 
•under-claimers. The present definition uses information available far 
all applicants and allows distinctions to be made between kinds of 
serious misreporters . 

5.6 Predictor Variables 

Sixty-six predictor variables that were used in the analysis are 
presented in Exhibit 5.2 with asterisks marking the ones that contribute 
to defining the model in its final form. Note that some splits could 
have been defined by more than one variable (e.g., Father's portion' is 
blank or zero could be defined by variables 15 or 17). The variables are 
in no particular order. 

5.7 Strengths and Limitations ' 

One of the msjor limitations of this approach to error-prone modeling 
is that one can nevt,^ be certain of having produced the best possible 
solution. A different investigator,, using different samples, different 
variables, or forcing a different first split could well come up with a 
totally different model. What could have been an effective split at the 
second .-eration may never appear in light of a different split at the 
first. 

THAID has a tendency to select splits that come close to dividing a 
sample or subsample in an uneven division. For this reason criteria 
which define only a very small percentage of the population would be 
un.likely to appear in the model. The one tenth of one percent of the 
applicants defined by a given PEC could be expected to be scattered among 
the thirty-seven groups resulting from the study. For this reason the 
model should not be used to the exclusion of other criteria that have 
also . -oven effective. 
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EXHIBIT 5.1; DEFINITIOM OF DEPETOT VAIIIABLE 
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E'XHIBIT S.2: PREDICTOR VARIABLES 



1. Citizenship (citizen, resident or blank) 
Z. Bachelor's degree (no or blank) 

Tax filing status (based on tax form, estimated, did not have to 

file or blank) 
4, Number of prior transactions 
*5/ M.D.E. source (SACC was coded the same as CSS) 

6. Marital status 

7. Discrepancy between household size and exemptions 

8. Household size 
Exemptions 

10, Marital status 

U. Social Security Benefits 

12. Mon-taxable- other than Social Security 

13. ' Non-taxable income 
*14. Adjusted Gross Income 

15. Father ' s portion 

16. Mother's portion 

*17. Presence or absence of both sources of earned income (both 

absent, father's only, mother's only, both present) 
*18. Taxes paid 

*19. Taxes as a proportion of AGI (blanks converted to zeros) 

20. Post-high school education (members of household in) 

21. Unreimbursed tuition 

22. Medical of dental expenses 

23. Casualty-theft losses 

24. Itemized deductions 
*25. Savings 

26. Applicant's resources 

27. Veteran's benefits (amount only) 
*28. House value 

29. House debt 

30. Investment value 

*31. Date first application signed 
*32. Data selection transaction processed 
33. Date of birth 
*34. Type of school (first choice) 
*35. Control of school (first choice) 

36. Presence or absence of second choice school 

37. Congruence between state listings (whether state of legal 
residence, state in applicant's address and state of first 
choice school coincide; one value for each possibility includina 
missing school ) ^ 

38. Number of blanks and zeros in fields calling for dollar 
responses 

*39. Eligibility index 

*40. Total income (N.T.I. + A.G.I. + amount received from veteran's 
benefits in a year after converting blanks to zeros) 
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SXHr3IT 5.2: PREDICTOR VARIABLES (CONTD.) 



*41. Yeop in school 

42. iVumber of assumption comnents 

43. Unusua] e;(pi?nses (Medical or dental + casualty-theft losses) 
*44. Assets (Hou.,e, farm, business and investment values miSSf^ 

debts, plus savings and applicant's resources) 

45. Father's portion divided by total income 

45. Total income divided by (total income. + assets) 

Ao' iPP^ -cant's resources divided by parent's total income 

values fslvingsr ''''''''''''' ^''"^ ^"^^■"^s^ 

^49. Non-taxable income divided by total income 

tiif pasf '"^'^'""^ "^^^^ ^^^^'"9 -received rejection comments in 

51. Number of changes made to critical fields 

52. Number of verifications in selection transaction 

b^. Number of fields producing assumption cofiments across 
transactions 

54. SEI difference between immeidata prior transaction and oresent 
transaction {SEl was computed if previous transaction was not 

V3 110/ 

55. Previous ineligible transaction? ■ 

56. Ever changed marital status?' 

57. Ever changed household size? 

58. Ever changed U.S. tax figures? 

59. Ever changed exemption? . . 
. 50. Ever changed AGI? 

51. Ever changed taxes paid?-. 

52. Ever changed model? ■ 

63. Ever a change in scheduled award? 

tr^Vjf,l^ y?^"? difference 'between highest and lowest SEI 

achieved (included computed for rejections) 
65. Number ever missing among the following fields: bachelor's 

^'^^l I'igu^es. exemptions. A.G.I. , 

. J ^^^^ Signed, year in school 

chose by"THAIDr^^^ division of sample, but not 
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method a so y-felds rpcrnUc u nucnuiai se^rcn 

^ --J gr.at.r variety or re.Tiedial actions-. 
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6 

THE ERROR-PRONE MODEL 

^•^ i^esults of the Validation and NonvalidatTon Sample Comoari^nn,, 
This section presents the overall findings of the analysis with 
respect to the validation and nonval i.dation groups. The general findings 
corroborate those reported in Part r of this report, namely that 
selection for validation is accompanied by different corrections behavior 
from the rest of the applicant population. 

Among random validation anplicants included .in the working sample, 
3.9 percent were ovsr-cl aimers (i.e. made corrections to their SER 
resulting in at least a 50 point increase in Scl from selection to 
criterion transaction), 4.5 percent were under-claimers and 35.6 percent 
failed to ra-enter the system (these figures were calculated weighted 
dependent sample). The corresponding figures for the replication samples 
were very similar, at 8.7 percent, 4.4 percent, and 36.1 percent, 
respectively. 

For the nonvalidation sample, however, only 1.3 percent corrected to 
their disadvantage and 0.7 percent corrected to their advantage. The" 
percentage failing to re-enter the system was 31.2 percent. In other 
words 20.3 percent of applicants who re-enter the system after selection 
for validation make corrections affecting their SEI by 50 points or more, 
but only 2.9 percent of those not selected for validation do so. 
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6.2 Description of Model Results 

This section presents a global description of the error-prone model, 
highlighting its most important features and discussing the major 
findings associated with the model. 

Thirty-seven groups, 25 dependent and 12 independent were identified 
through the sequential search procedure. Twenty variables suffica as 
definers of the 37 groups: 

• one relates to corrections behavior (SEI change), 

• five are algebraic manipUl ations of SHR fields (SEI, total 
income, assets, taxes as a proportion of AGI, and NTI as a 
proportion of total income), 

t one is a logical statement relating two fields (presence or 
absence of each portion earned, with four possibilities), 

• two are dates (date application signed and date transaction 
processed), 

two are related to school choice (type and control), 

one is related to application form. (MDE source), 

One is answered indirectly by the applicant (model or dependency 
status), and 

• seven are answered directly in the application (AGI, savings, 
house value, tax filing status, taxes paid, exemptions and year 
in school). 

Exhibits 6.1 and 5.2 provide two different ways of viewing- the 
model.' Exhibit 6.1. presents the tree indicating the splits that led to 
the 37 groups. Letters represent nodes (subsamples which were 
subsequently split) and numbers represent the 37 groups. As can be seen, 
eight splits were the most that were necessary to create the most complex 
groups, while three sufficed for the simplest. 

Exhibit 5,2 simply lists the defining characteristics of each group. 

Before considering the misreporting patterns of the different groups, 
the reader should be made aware of certain trends in the splits producing 
the model. Applicants whose MDE source was BEOG had similar patterns to 
those whose MDE source was PHEAA, while ACT applicants resembled CSS 
applicants (SACC applicants were coded as if they were CSS). ACT and CSS 
applicants produced more splits (and therefore more groups) than did BEOG 
and PHEAA applicants. Twenty groups were composed entirely of ACT and 
CSS applicants, seven of BEOG and PHEAA applicants, and ten had 
applicants from all four sources. 
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EXHIBIT 5.1 (CCNT.) : NODES IN TREE DIAGRAM 

Node : Definer of Latest Split 

A All applicants 

8 Dependents 

C Independents 

D Estimated taxes 

E Used tax form, did not have to file or left space blank 

r 3E0G or PHEAA 

G ACT or CSS 

H SEI not over 400 

I SEI over 400 

J SEOG or PHEAA 

K . ACT or CSS 

L SEI « 0 

M SEI greater than 0 

N Savings 0 or blank 

0 Taxes paid not over S2,000 

P Taxes paid over S2,000 

Q House value « 0 or missing 

R .House value greater than 0 

S SEI not over 200 

T SEI over 200 " 

U Exeinptions = blank, 0 or 1 

y Exemptions = greater than 1- 

W Processed by April 30- 

X SEI never has changed 

Y Both parents ' positions are greater than zero 
First choice school is university or blank ■ 

AA Total income less than or equal to 510,000 

SB Taxes less than 5% of A6I or both figures zero 

CC Taxes over 5%'of AGI 

00 Processed by April 30 

EE Processed after April 30 

FF Taxes paid not over $500 

GG SEI less than or equal to 1200 

HH Taxes paid greater than zero 

II Taxes over 15% of AGI 

JJ SEI not over 600 
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EXHIBIT 6,1 fCONT,) : GROUPS IN TREE DIAGRAM 



Group ; Definer of Latest Split 

1. Father's portion = 0 or blank 

2. Father's portion greater than 0 

3. Savings greater than 0 

4. Processed by May 31 

5. Processed after May 31 

6. SEI changed at least once 

7. First choice school is university 

3. First choice school is blank or not university 

9. SEI greater than 1200 

10. At least one parent's portion is zero or blank 

11- First choice school control is blank or public 

12. First choice school is private or proprietary 

13. 6E0G 
U. PHEAA 

15. First choice school listed, but not a university 

16- NTI less than or equal to 50% of total income (or both zero 
or one negative) 

17. NTI over 50% of total income 

18. Total income over S10,OOO 

19. Taxes paid = blank or 0 

20. SEI not over 600 

21. Assets not over $30,000 

22. Assets over 530,000 

23. Taxes over 15% of AG I 

24. AG I not over $25-, 000 

25. AG I over $25,000 

26. SEI 0 

27. SEI greater than 0 

28. AG I not over 52,000 

29. AG I over S2,000 

'30. Processed by July 31 

32. Yev in school blank or 1 

33. Year in school 2,3,4 or 5 

34. Signed by February 28 or date missing 
35 • Signed after February 28 

36* Taxes paid over $500 

37 • Processed after April 30 
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EXHIBIT 6.2: OEFINITIOM OF THE 37 GROUPS 



Groups 1-10: Dependents whi< »sLi"i»:.H ♦ ixes 

1. SEI not over 400, nc .savings or father's portion. 

2. SEI not over 400, no savings, but father's portion greater than 0. 

3. SEI not over 400, savings greater than 0. 

by^MarSl'^^^' ^^^^^ ""^ ' changed, processed 

5. SEI over 400, taxes not over S2,000, SEI has not changed, processed 
atter May 31 • 

i' c^5 ^^^^^ S2,000, SEI has changed. 

7. Scl 401-1200, taxes over S2,000, both portions greater than 0, first 
choice IS university. ^ , 

8. SEI 401-1200, taxes over S2,000, both portions greater than 0, first 
choice missing or not university. , i=.i, 

in* III ^^^^^ 52,000, both portions greater than 0. 

10. SEI over 400, taxes over 52,000, one or both portions is 0 or blank. 

Groups 11-15: Dependents, did not estimate, MDE is BEOG or PHEAA 

11. House value 0 or blank, first choice school public or blank 

n* Snp'f.'RJnr ^^'"^ ^^''^ P''^'^^^^ °^ proprietary. 

13. I'lOE is BEOG, house value greater than 0, university or no first 
choice. 

14. MDE is PHEAA, house value greater than 0, university or no first 
choice. 

15. House value greater than 0, first choice school is listed, but not a 
university. , u^-. ■ ul a 

Groups 16-25: Dependents, did not estimate, MDE is ACT or CSS 

16. SEI not over 200, total income not over 510,000, NTI is not over 50% 
of total income. 

17. SEI not over 200, total income not over 510,000, NTI over 50% of 
total income. 

18. SEI not over 200, total income over 510,000. 

19. SEI over 200, taxes paid 0 or blank. 

20. SEI 201-600, taxes greater than 0, but not over 5% of AGI. 

^i. iti over 600, taxes greater than 0, but not over 5% of AGI, assets 
not over 530,000. 

n?L°«n nSS' ^^^^^ greater than 0, but not over 5% of AGI, assets 
over 530,000. 

23. SEI over 200, taxes 5 to 15% of AGI. 

24. SEI over 200, taxes over 15% of AGI, AGI not over 525,000. 

25. SEI over 200, taxes over 15% of AGI, AGI over S25,000. 
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EXHIBIT 6.2: DEFINITION OF THE 37 GROUPS (CONT.) 

Groups 26-27: Independents, MDE is 8E0G or PHEAA 

26. SEI = 0. 

27. SEI greater than 0. 

Groups 28-37: Independents, MDE Is ACT or CSS 

over's2;oOo!""'^^°"' ""^ ^ °' ^'^""^''^^ -^P^^l 30. AG! not 

ilooo?' ^'"^ ^ °^ ''l^nk. processed by April 30, AGI over 

51 ' " S' are 1 or blank, processed May 1 to July 31. ■ 

32 In : are 1 or blank, processed after July 31. 

33* 111 I n' ^'"'J^' ^' y^^*- ^" school is 1 or blank. 

34 Wl ^' y^^*- school greater than . 

• s?^nL^\?feS?u"ar?'2n?^5:?e^ ^^s??^^ '^'^^ °- 

• s?gn^5^^TerV^'^r2ar^a!"'' ''''' 

37' spf J!;^" S' P''o*=^"ed by April 30. taxes c/er S500. 

J7. SEI greater than 0, processed after April 30. 
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T-ie and control of institution contributed to the definition of all 
five dependent 3E0G and PHEAA groups, and of two of the groups including 
all four sources. Type and control of institution did not seem to make a 
difference for ACT and CSS applicants, since they entered into the 
definition of none of the twenty groups composed entirely of applicants 
from these two sources. 

•Aside from SEI (which split the independents), information related to 
tax forms was not at all useful in predicting misreporting for 8E0G and 
PHEAA applicants. Dates of processing and- signing also failed to predict 
for this subsample. On the other hand, SEI, dates {for independents) and 
information that could be verified directly from the tax forins (AGI , 
taxes paid, exemptions, etc.) proved to be the best predictors for ACT 
and CSS applicants. Among applicants who estimated taxes (the ten groups 
containing applicants from all four sources) EI, portions (presence or 
absence) taxes paid and. savings prcved the best predictors. 

Table 6.1 shows the distributions on the dependent variable for the 
two random validation and thenonvalidation samples. The column marked % 
lists the percentage of the total sample represented by the group. The 
other four columns represent the percentage of each type in each group.. 

As can be seen,' .Group 7 has the highest percentage of overclaimers, 
but most of the groups made up of dependents who estimated taxes are also 
relatively high. Some of these (like Groups 2 and 3) have four and five 
times as many over-claimers as under-claimers, while others (like Groups 
4 and 6) have approximately equal numbers of both types. 

Group 24, characterized by a high tax rate for the income of its 
members, is the only group among dependents who did not estimate taxes 
with a high proportion of over-claimers. Group 21 has the highest 
proportion of under-cl aimers. 
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Independents had proportionately fewer tnisreporters, but there wero 
some independent groups with a large proportion missing and a relatively 
high rate of misreporting among those who did re-enter the system. The 
missing rate could not be attributed to a deterrent effect of validation 
since It was generally replicated in the nonvalidation sample It is 
possible that the change in SEI formula (which would have affected aiany 
of these applicants) could have led financial aid administrators to 
recommend that applicants wait until they received the systeoi generated 
SER.. If so, they aiay be delaying their re-entry. Alternately, some may 
have felt their grant was too small (under the old formula) and made 
other plans. 

While no nonvalidation group had more than 7 percent making ' 
corrections which raised their SEI (the eouivalent of over-claimers for 
the validation groups) some of the groups with the highest proportions in 
nis category were also groups with high percentages of over-claimers 
In other words, validation groups with high corrections rates tend to' 
correspond to nonvalidation groups with relatively high proportions of 
spontaneous corrections, but the rates of corrections among the 
nonvalidation groups are much lower than those of the corresponding 
validation groups.' 

>^PPTi cants W ho Did Not Re-enter the 'System 

In order to examine the extent to which applicants may have been 
intimidated by 'selection, the percentage of missing applicants for each 
nonvalidation group was subtracted from the percentage of missing 
applicants in one of the validation groups (whichever yielded the most 
conservative results). Table 5.2 presents this information. Four groups 
yielded results over 10 percent, two of these (Groups 2 and 24) had' high 
rates of ov.r-claimers. The third group (Group 9) combined estimating 
taxes with a high SEI, possibly leading its members to decide that 
validation was not worth the trouble since the award would have been 
ow. The fourth c,roup (Group 31) had selection transactions processed in 
August, and the delay in getting validation materials together may have 
made these applicants miss the first update of the recipient file 
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One group had significantly fewer missing applicants when selected 
for validation than otherwise. This group is the one most likely to have 
been affectad by the formula change in the sense that their old £1 would 
have been high, while their new one would be zero. Seventy-five percent 
of this group's nonvalidation applicants failed to re-entar the system, 
as opposed to 59 percent of the ones in which validat-on sample. In this 
case validatdon may have been an incentive to remain in the system. 

When one compares the figures in the first column with percentages of 
misraporters for non-missing cases.no clear pattern emerges. There does 
seem to be a greater tendency, for validation applicants to be missing, 
but the reasons are probably varied and difficult to determine. 
6.4 Misraportinq by Critical Field 

As was stated in Part I, the fields most frequently changed after 
selection were NTI, AGI and taxes paid. Table 6.3 reports the percentage 
of non-missing applicants whose SEI and selection transaction values in 
these fields were too high or too low. The average amount by which each 
of these fields was misreported is given in Table 6.4. 

An examination of those groups with high percentages underreporting 
NTI reveals that they are not groups with large numbers of serious 
misreporters. Groups U and 17 had 20.3 percent and 26.4 percent 
underreporting their NTI, but their percentage of over-claimers (counting 
only non-missing applicants) were only 7.5 percent and 9.3 percent 
respectively, well under the average of 13. S percent. Only for Group 19 
does NTI seem to be a possible source of serious error, and here it 
affects both over-claimers and under-cl aimers. 

AGI is another matter altogether. Group 8 had 50 percent of its 
non-missing members misreporting AGI to their advantage (though — one 
of them was found to have underreported NTI). Group 4 had 71.4 ^ent 
of those applicant who re-entered the system making corrections to AGI 
after selection. Table 11-4 shows that the mean discrepancy is over 
S1,000 for seven groups, and this includes all applicants who re-entered. 



6.11 



139 



TABLE 6.2: EFFECTS OF VALIDATION ON RE-ENTERING AND ON SEI CHANGE 



Group 

I 
2 
3 
4 
5 
5 
7 

a 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 

37 



Difference In 
Percentage Missing* 
( Va 1 i dat i on-iVon -va 1 i dati on ) 

3.5 
13.0 
1.2 
-.03 
-.04 
4.3 
-4.5 
-0.9 
12.3 
5.9 
-5.7 
5.4 
-1.7 
1.8 
3.6 
4.0 
6.0 
8.7 
8.1 
2.6 
1.1 
4.5 
6.2 
12.0 
0.4 
3.8 
4.4 
-2.9 
-16.2 
5.6 
11.6 
5.5 
1.7 
-4.6 
-0.8 
. -0.6 
1.0 



Serious 
Misreporters as 
Percentage of 
Non -missing** 

17.9 

38.0 

33.7 

56.4 . 

50.4 

34.7 

53.4 

31.5 

28.1 . 

51.1 

11.5 

15.4 

20.1 

37.9 

18.2 

11.9 
9.6 

16.9 

41.9 

16.2 

43.2 

23.9 

19.0 

47.9 

24.6 

4,3 
35.1 

3.S 

9.9 

5.0 

3.4 

5.8 

4.8 
52.0 
28.6 
61.5 
25.0 



Total 
Misreporters 
As Percent of 
Non -missing*** 



Most conservative estimate, validation minus nonvalidation. 
^ II °t SO points' or more, working sample. 

SEI change of 1 or more points, working sample. 



5.12 



.9 
.5 
.'9 
.9 



25.9 
56.0 
44.7 
80.0 
65 .3 
42 
70 
51 
46 

. 63.0 
17.2 
19.2 
25.1 
47.0 
22.5 
16.1 
14.4 
25.3 
49.7 
25.0 
51.6 
37.0 

27.5 

60.6 

29.3 
4.4 

38.8 
4.4 

11.9 
5.6 
5.1 
6.2 
5.3 

56.0 

28.6 

65.4 

25.6 
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TABLE 6.3: 



PERCENTAGE'MISREPCRTING MAJOR FIELDS 



Group 

1 
2 
3 
4 
5 
6 
7 

a 

9 

10 

11 

12 

13 

14 

15 

16 

17 

13 

20 

21 

22 

23 

24 

25 

26 

27 

23 

29 

30 

31 

32 

33 

34 

35 

36 

37 



Too 

High 

5.0 

1.0 

5.3 

4.3 

4.7 

8.8 

0.0 

2.2 

0.0 

1.5 

6.6 

10.0 
6.8 
0.0 
5.0 
1.6 
8.9 
3.6 

13.2 

2.4 

9.5 

8.7 

1.4 

1.4 

0.0 

4.6 

5.0 

4.7 

0.0 

3.1 

1.7 

5.1 

3.1 

4.0 . 
11.9 

3.8 

4.0 



NT! 



Too 
Low 



15.1 
6.0 
15.2 
5.6 
5.5 
4.8 
1.1 
0.0 

o.o 

3.7 
20.3 
8.5 
10.5 
16.7 
10.4 
12.4 
26.4 
11.0 
18.0 
2.4' 
11.6 
3.3 
2.6 
1.4 
0.8 

12.0 
5.0 
7.6 
2.0 
6.2 
6.8 

13.9 

11.0 

10.0 
7.1 
3.8 
5.2 



Too 
High 

9.4 
20.0 
12.1 
24.4 
15.0 

7.5 
18.2 

7.6 

7.3 

8.9 

4.8 

2.3 

3.7 

6.1 

2.9 

4.5 

2.1 

4.5 

9.0 

4.8 

7.4 

8.5 

3.9 

5.3 

3.3 

3.0 

6.8 

1.5 

6.9 

2.2 

8.5 

2.2 

3.6 

8.0 

7.1 

7.7 

2.2 



AG I 



Too 
Low 



Taxes- 
Too Too 
High Low 



SEI 

Too Too 
High Low 



26.6 
38.0 
17.7 
47.0 
35.4 
16.3 
39.8 
50.0 
31.3 
38.5 
6.2 
8.5 
8.2 
7.6 
6.1 
14.7 
6.7 

13.0 
9.9 
5.4 

14:7 
6.5 

10.3 

23.9 

12.3 
8.9 

10.4 

18.7 

20.8 

13.0 
6.8 
8.3 
8.4 

24.0 
2.4 

26.9 

9.0 



8.6 

16.0 
13.3 
27.8 
19.7 
3.2 
22.7 
21.7 
17.7 
27.4 
5.3 
2.2 
5.9 
9.1 
5.9 
6.1 
0.2 
7.8 
0.0 
5.4 
.3.2 
7.6 
8.1 
40.8 
12.3 
3.8 
5.4 
5.8 
17.8 
4.0 
5.1 
1.8 
2,6 
8.0 
0.0 
30.8 
7.1 • 



10.1 
26.0 
15.5 
35.9 
33.1 
13.6 
36.4 
32.6 
25.0 
22.2 
2.2 
6.2 
3.7 
10.6 
2.3 
6.4 
1.1 
7.5 
10.8 
6.0 
22.1 
9.8 
5.1 
5.6 
2.3 
2.0 
9.0 
3.2 
6.9 
4.0 
O.O 
2.2 
2.1 

18.0 
0.0 

11.5 
5.2 



7.9 
11.0 
11.7 
36.3 
29.1 

21.8 

21.6 

13.0 

17.7 

14.1 
5.7 
7.7 
8.2 

12.1 
9.0 
2.2 
2.5 
5-. 5 

27.5 
. 12.6 

33.7 
23.9 

9.4 

8.5 

6.2 

0.0 
22.1 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 
26.0 
16.7 
11.5 

9.3 



18.0 
35.0 
33.0 
43.2 
36.2 
21.1 
48.9 

• 48.9 
29.2 
48.9 
11.5 
11.5 
16.8 
34.8 
13.5 
13.9 
11.9 
19.3 
22.2 
U.4 
17.9 
13.0 

18.1 

52.1 

23.1 
4.4 

16.7 
4.4 

11.9 
5.6 
3.1 
6.2 
5.3 

30.0 

11.9 

53.8 

16.4 
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not just those who changed the field. (The average discrepancy for Group 
2 was S2,230, but if one counts only applicants who corrected this field, 
the average correction was S3, 345 in either direction.) A6I is therefore 
one of the fields most frequently changed, and probably the one causing 
most errors. It is also one of the easiest items for financial aid 
administrators to validate. 

Taxes paid, which is also easy to validate, was a high srror rata 
field. Among Group 24 applicants, 40.3 percent of those not missing 
overrsported their taxes (particularly disturbing given that- they claimed 
to not have estimated). Group 4 had the largest proportion of 
ntisreportars in this field (62.7^). 

Changes in AGI were not regularly accompanied by changes in portions 
earned. With the exception of Group 2, 41 percent of which changed 
father's portion, the percentage changing portions in any group. seldom 
reached 20 percent. 

6.5 Impact of Mis reporting 

Table 5.5 presents a breakdown of EI and scheduled award 
discrepancies between criterion and selection transactions. As can be 
seen, for some groups (e.g. Group 5) over-claimers and under-cl aimers 
cancel each other out, even, though the average award may be off by over 
SIOO. For other groups (a.g. Group 10) over-claimers far outweigh 
under-claimers produciiicj a mean net differences of S76 per applicant. 
Assuming that expected disbursement is 94 percent of scheduled award (as 
was the case for the 1978-1979 academic year) and that the proportion of 
non-missing applicants represented by this group is the same as its 
irepresentation in the recipient file the amount saved by validating all 
Group 10 applicants, over validating none would be over $2,000,000. This 
is not a net-savings amount, since it does not take into account the fact 
that some are being validated already, or that a deterrent effect might 
produce additional savings. 

Group by group recoinnendations and interpretations are presented in 
Chapter 7 and implications for validation are suggested in Chapter 8. 
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TABLE 6.4; 



MEAN NET AND ABSOLUTE DISCREPANCIES IN CRITICAL FIELDS 



Group 

1 
2 
3 
4 
5 
6 
7 
8 

. 9 ■ 

10 ■ 

11 

12 

13 

14 

15 

16 

17 

18 
. 19 

20 

21 

22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35. 

36 

37 



NT I 

Net* Abs. 



179 
87 
325 
-174 
-18 
-448 
22 
-4 
0 

131 

427 
-8 
119 
477 
174 
350 
473 
264 

-as 

18 
-193 
-122 

-11 
-117 

22 
177 
-27 

72 

13 

90 

89 
326 
198 
-32 

22 

16 
-56 



614 
134 
487 
282 
184 
548 
22 
4 
0 
148 
708 
359 
301 
477 
363 
397 
769 
34S 
773 
87 
633 
285 
125 
239 
22 
263 
164 
174 
13 
140 
124 
382 
300 
187 
172 
76 
154 



N-St* 

329 
1,230 

455 

421 

264 
98 

802 

491 

333 

567 

242 
62 
95 
-279 

-57 

670 

256 

<156 

iOO 

17 
-92 
-1S4 

42 

85 
168 

86 

49 
301 
271 
390 
-61 • 
172 
298 

88 
-48 
294 
125 



AG I 



Abs. 

591 

2.230 

1.002 

1,493 

1,143 
608 

1,193 
979 
516 

1,155 
56G 
373 
229 
421 
353 
855 
320 
615 
583 
201 
564 
250 
253 
1,145 

2076 
172 

312 

317 

370 

413 

181 

25T 

409 

457 
85 

306 

236 



Taxes 
Net* 

13 

103 

41 
166 
230 

56 

68 
-67 

73 
-173 
-26 

35 
5 

-8 
-57 
15 
10 
-20 
144 
17 
280 
98 
-8 
-312 
•128- 
-13 
16 
7 
0 

27 . 
-15 
2 
15 
62 
0 

-83 
4 



Abs. 

45 
202 
128 
366 
355 
110 
298 
288 
239 
426 
39 
49 
49 
154 
83 
81 
11 
142 
144 
■ 42 
296 
117 
69 
-147 
134 
22 
36 
19 
60 
40 
15 
17 
26 
100 
0 

108 
49 



Net discrepancies are averaged without talcing the absolute value 
overreporters and underreporters could cancel each oth«r out. Se 
value is subtracted from criterion value. 



so that 
lection 
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TABLE 6.5: MEAN NET AND ABSOLUTE SEI AND AWARD DrSCREPAMCIES 



Group Net* ,,,, ' ,,,,, ^^-^ 

2 II ■ 12 21 

3 7I 28 35 

4 It 46 53 
.J 25 lai ^"^ 



5 -15 u 1^ 125 

7 .2 32 



29 ■ ^0' 55 % 

92 . 1^? 



12 a ifl ,„ 20 

\l ^ « ff " 



1 ^ 21 12 13 

» 1 I 

1^ - I -1 

- s I i 

II 13 g 



i " » SI 



* Criterion-selection. 
** Selection-criterion. 
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DESCRIPTION AND INTERPRETATION OF THE GROUPS 

This chapter describes each group in terms of its .nisreporting 
patterns and selection transaction profile. An attempt will be made to ' 
interpret ana identify possible reasons for misreporting and to suggest 
corrective action. Each description should be' read in conjunction with 
Tables 6.1 through 6.5. 

At this point a few terms should be reviewed. The term over-claimer 
refers to an applicant whose SElWas at least 50 points lower on the 
selection transaction than on the criterion transaction. Under^cT aimer 
refers to an applicant whose SEI was at least 50 points higher at 
selection than at criterion. Mis reporter includes both over-cl aimers and 
under-cl aimers. The net discrepancy on a given field between criterion 
and selection transaction refers to the average of the figure at 
criterion minus the figure at selection. Absolute award difference and 
absolute discrepancy on a given field refers to the average of the 
absolute value of the figure, at criterion minus the figure at selection. 
Gains and losses may cancel each other out In computing net discrepancy 
of net award difference, but will not in computing absolute award 
difference or absolute discrepancy. 

'?'"°."P. y- dependents who estimated taxes, had SEI not over 400\ no 
savings and father's po rtion mission or 0 . This group had the lowest 
rates of misreporting among the ten groups of dependents who estimated 
taxes. They are a low income group, where the mother's earned income 
accounts for most of the AGI. This group had the largest percentage of 
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corrections to mother's portion (26.6/,). but its rate of misreporting 
identified through validation was below that of the total sample. On the 
other hand, since this group consists largely of applicants with divorced 
or separated parents the possibility of an applicant providing the wrong 
definition of parent (i.e., listing their mothers when they lived a 
longer period with their fathers) is a real one that validation may not 
identify. Quality control studies might wish to investigate this issue. 
This group should not be selected for validation, but should be 
incorporated in any remedial action pertaining to estimation of ta.xes. 

^i^2ii£j.: dependents w ho^ estimated taxes, had SEI not over 400, no 
savings and father's po rtion greater than 0 . This group differs in its 
defining. characteristics firom the previous one only in that father's 
portion was present. Its misreporting pattern, however, is very 
different. Over-claimers constitute 19.3 percent of the applicants in 
this group, and 32.1 percent of those not ,iiissing. Furthermore, this is 
one of the four, groups with many more missing cases in the validation 
samples' than in the nonvalidation one (UM and 13.0% respectively). 
Corrections to AGI were made hy 58 perct itof the nonmissing applicants 
in this group, and the mean correction iv.r. ^r,r 33,345 in one direction 
or the other. This was the largest amount any group, resulting in a 
net underreporting of AGI of Sl,230 (calculated among all non-missing 
applicants). On the other hand, this did not translate into very large 
discrepancies in SEI or award due to low income and assets. 

Several possibilities exist. Since thi . group was mors likely to 
over-claim than under-claim, the suspicion of some intentional 
misreporting and some failure to re-enter the system out of fear of 
getting caught must be entertained. If this is the case some other 
fields which are harder to validate, such as- the absence of savings, 
could also be sources of Qrror for this group. On the other hand, this 
could be a case of low income applicants trying to estimate their income 
carelessly and feeling intimidated by the validation requirement. The 
group should probably be selected for validation in the a.bsance of 
remediation concerning estimation. Even if verification of tax returns 
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is required, for all who estimate, at least part of this group should be 
validated with view toward possible errors in fields unralated to tax 
returning or failure to report assets. 

group 3: dependents who estimated taxes, had SEI not over 400 and 
had, savings. The presence of savings differentiates this group from the 
previous two. While the proportion of over-claimers among all applicants 
in this group Is. slightly higher, the proportion of misreporters among 
those not missing is lower than that of the previous group. (This Is due 
to the absence of the extra 13 to 13 percent which failed to re-entar in 
the previous group when selected for validation.) Nevertheless, the 
percentage of over-claimers is inordinately high, reflecting the fact 
that the attempt to estimate income and ta:<es is, for the most part, 
likely to produce errors. The absence of additional missing cases when 
compared with the nonvalidation sample, and the fact that savings were 
reported suggests honest mistakes rather than fraud. The group should 
probably be selected for validation in the absence of remediation 
connected with the estimation of taxes. 

.^■'"P"P . dependents who estimated taxes, had' SEI over 4Q0. ta.xes not 
over $2.0 00. no change in SEI prior to selection and were processed bv 
'^?y 31. This group has the highest proportion of mi'sreporters 
(over-claimers and under-claimers combined) of any in the model. Fully 
80 percent of non-missing applicants changed their SEI after selection, 
56.4 percent of non-missing applicants doing so by 50 points or more. 
Among these applicants 71.4 percent corrected A6I and 63.7 percent, 
taxes paid. While this group misreports to a great degree, it is almost 
as likely to hurt itself as help Itself by its misreporting. It even 
does some unsolicited correcting among nonvalidation applicants. This 
group should' be selected, for validation in the absence of other 
corrective action. 

Groue_^: dependents who estimated taxes, had SEI over 400, taxes not 
over $2,000, no change in SEI prior to selection and were processed after 
Maj^. The total rate of misreporting for this group is slightly lower 
than for the previous one, it is still among the highest. There is also 
less unsolicited correcting among the nonvalidation applicants. It seems 
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less excusable to estimate taxes when one files after April 15 than when 
one files before, and it is with this sort of group that a change in 
procedures requiring verification with tax forms is 1n order. Falling 
some other sort of remediation, this group should be selected for 
validation. 

^^CmJ.' dependents who estimated taxes, had SEI over 400, taxes not 
oyer S2,000 and a chang e in SEI from a previous transaction . While still 
including more misreporters than the total average, this group is lass 
error-prone than the two previous ones. Selection of this group is 
probably not indicated unless one were to select a large proportion of 
applicants for validation. More to the point is the, fact that this group 
could be eliminated in most cases through corrective action. By the time 
an applicant receives an SER it is late enough in the year that he can 
verify his figures from his tax forms. The requirements that applicants 
do this and sign a st.^tement to the effect prior to submitting 
corrections would completely eliminate applicants with this set of 
characteristics, and presumably reduce their misreporting. 

^r°"P ,7/ dependents who estimated taxes, had SEI from 401 to 12QQ 
taxes over S2,000, both portion. < ^ rp ater than 0 whose first choice is a 
""^''^^^^^y .- This group has the largest percentage of over-cl aimers of"^ 
all thirty-seven groups. It is a high income group to begin with 
(average A6I reported at selection is S25.536), and its manbers seem to 
be misreporting to their advantage. The average member of this group who 
re-enters the system gets S68 less after selection for validation than 
before. The average award is off by S99 (there are 9.6 percent 
under-claimers). In spite of this tendency to over-claim, this group 
also has the lowest proportion missing in both validation samples. AGI 
is corrected by 58 percent of its non-missing members and tixes paid by 
59.1 percent. Validation seems to be in order for this group, unless an 
alternative means of handling the errors due to estimation of taxes is 
found. 

^^3EJ.: dependents w ho estimated taxes, had SEI from 401 to tznn, 
^axes over $2,000, both p ortions greater than 0. whose first choice 
inst'itution is missing or not a university - This group has a lower rate 
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of over. c1 aimers than the previous one. but still one of the highest 
rates of any group. Actually, a higher percentage understate AG! 1n this 
group (SO?S) than in any other group, but the amount of the discrepancies 
are not as large as 1n Groups 2 or 7. This group should also be 
validated. 

de pendents who estimated t..xes. had Si: ovpr 1200. taxes 
oyer $2,000 and both portion, greater than 0 . This group had 
misreporting rates higher th.n the total sampla, but lower than rnost 
applicants who estimated taxes. The fact that their award potential was 
low at selection limits the possible Impact of validation. The group 
Should not be validated, but corrective action with respect to estimation 
of taxes could be considered. 

^P .' dependents who estimated ta xes, had SEI nv er 400. taxe<; 
over S2,000 and at least on e p ortion 0 or blank . This is another group 
with a high rate of over-cl aimers . It has the highest net award 
difference from selection to criterion (S76 per applicant overpayment at 
selection). It was estimated in Chapter 4 that the difference between 
.paying all recipients who fall in this group at selection and paying them 
after validation or other corrective action taken. 

■ '^'^ ■ °"P . ^l ' dependents wh o did not estimate tax figu res, filed thmngh 
8EQG or PHEAA, .did not own a house and f: h ose a public institution or non^ 
at_all, This group has very low rates of misreporting and a high rate of 
missing. The percentage missin-) is even high-, among nonvalidation 
applicants, so validation coulo even be ac::>:. an Incentive to remain 
in the system. The group should not be va • -: .-, nor is corrective 
action needed. 

Pr9"P .. 12 : dependents wh o did not estimate tax figures, filed through 
BEOG or PHEAA, did not own a house and their first choice school was 
private or proprietary. Similar to the previous group, but the 
percentage missing is lower. There is some evidence 6f validation 
keeping some applicants away, but the figure is small enough to be 
possibly due to sampling error. In any case applicants in this group 
should not be validated. 



7.5 



149 



^^^mJl' dependents who did not estima te tax fioures, filed throug h 
SEOG, house value was great er than 0 and chose a univp,..ii-y nn 

institution . This group's misreporting pattern is close to average, and 
it should not be validated. 

-^1=2MBJ4: dependent ap plicants who did not estimate taxes, filgd 
through. PHEM. house value was greater t han 0 and fir^t choice was a 
university or no institution . This i. another case of a group with high 
proportion of missing cases across both validation and nonval idation 
samples. This is not surprising, since a large proportion of PHEAA 
applicants ars interested in form of aid other than Basic Grants. Even 
though the rate of misreporting is higher than the previous three groups, 
this group does not merit validation. 

^""""^P dependent applicants who did not estimate, filad through 
BEOG or PHEM, house value was greater th.n p and listed a first choice 
institution other than a univprsi>y. This is another group that 
resembles the total. Validation is not necessary. 

'^""'^P dependent applicants who did not estimate, filed through 
ACT or CSS, had Scl not over ? nn, tnt.i income not over $10. OOP and .nitt 
not over 50 percent of total TnrninP. While the rate of misreporting is 
.below average for this group, there are a few cases misreporting in large 
scale, resulting in the largest net discrepancy in AGI between selection 
and criterion for groups other than the ones estimating tax figures. The 
percentage correcting AGI is not large (19.2%), but the average 
discrepancy for those correcting is $4,453! Selective validation or 
tighter edits, including verification of AGI, msy well be in order. 

dependent app licants who did not estimat g, filed through 
ACT or CSS, had SEI not over 200. total income net n^s ar sn o.ooo and NT! 
over 50 percent of total income. This group has a lower than average 
rate of misreporting, in spite of the fact that 26.4 percent of its 
applicants who re-enter the system underreported NTI. This field 
constitutes the bulk of the income for this group, the lowest in AGI. 
However, in spite of the fact that NTI is underreported by $473 on the 
average, this error seems not to affect SEI much because of low income 
and merits at most an edit, but not validation. 
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Group IB: dependents who did not estimate, filed through ACT or CSS, 
had SEI not over 200 and total income over SIO.OQQ ." This group has a 
slightly below average rate of mi sreportTng, but also a few cases who 
substantially underreport AGI. Selective validation of AGI or even an 
edit requesting verification might suffice here. 

'^':^°"P ^g .- dependents w ho did not estimate, filed through ACT or CSS, 
had SEI over>200 and either nairi nn t ^xes or left taxes blank . This 
group had a moderately . high rate of misreporting, but it had more 
under-cl aimers than over-claimers. A good portion of this group's income 
is non-taxable and it had t, s. acond highest rate of NTI -changes after 
selection. There is some evidence of applicants not re-entering the 
system as a result of selection. Again, the rate of .Tiisreporting is not 
among the highest, but some of the few who do misreportdo so in a way 
that affects their SEI to a large degree. Among the 10.3 percent of the 
applicants who re-entered and corrected taxes the mean discrepancy was 
SI, 333. An edit requesting verifi cation of the tax field would be 
appropriate. Since this group receives on the average over S3,00O in 
Social Security benefits, these applicants may well find themselves in' 
unusual circumstances meriting the suggestion that they seek help in 
filling out their form. 

GrougJO: dependents who did not estimate, filed through ACT or CSS 
had SEI from 201 to 600, pairi t;,vpc, but- taxes were not over 5 percent of 
AGI_. This group . had a' moderate rate of misreporting, close to the 
average. Furthermore, the impact of the misreporting that does exist in 
this group is rather low. No corrective action seems to be required for 
this group. 

. ^"^""P ^1} dependents who did not estimate, filed through ACT or CSS, 
had SEI over 600, paid taxes, taxes were under 5 percent of AGI and 
assets were not over S30,000. This group had the largest proportion of 
under-cl aimers of any of the 37. Many of the under-cl aimers seem to have 
underreported taxes. The average applicant changing taxes paid after 
selection underreported by $1,107. This figure suggests the possibility 
of a digit being omitted. An edit pointing out the tax figure and a 
verification comment would be in order. 
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^'C°"P "^l: - dependents w ho did not estimate taxes, filed throug h ACT 
or CSS, had SEI over 6QQ . paid taxes, had taxg<; nn t, oyer 5 percent nf 
AGI.and assets over S3p,()00. Like the previous group, this group also 
has more under-claimers than over-cl aimers. The impact, however, is 
lower for this group. An edit or comment concerning taxes might be 
considered, but otherwise no remedial action is necessary. 

Groue_23: dependents who did not estimate taxes, filed through An 
or CSS, had Scl over 200 and taxes were 5 to 15 percent of AGI . This was 
the largest group, and its characteristics resemble those of the total 
sample. It is likely that further investigation, experimenting with 
other variables or other methods might discover further breakdowns of 
this group. To this end. the practice of random validation should be 
continued and the group .given special attention in future attempts to 
refine the model. In the interim, no special remedial action seems 
indicated. 

Grou2j4: dependents who d id not estimate taxes, filed throug h ACT 
orLCS^^'.ha d SEI over 200, taxes over 15 percent of AGI and AGI not nvpr 
S25^. This group had the greatest rate of over-claimers of any group 
which did not estimate taxes. Furthermore, the percentage of applicant 
who filed to re-enter the system- after selection for validation was much 
greater than the corresponding percentage in these applicants to fail to' 
re^^nt^r the system over the percentage in the non-validation sample. By 
and large, overreporting of taxes was the most likely source of error, 
but underreporting AGI was also prevalent. Two possible conclusions can 
be formed. First, one may assume intent of fraud, and increased failure 
to re-enter out of fear of getting caught. A second possibility is that 
after validation many applicants in this group (whose mean selection SEI 
is over 1,000) become ineligible and do not bother to file the 
correction. In either case, members of this group should be validated. 
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■ '^ ■ ^°"P dependents w ho did not estimate taxes, filed through ACT 
or CSS, had SEI over 200, taxes over 1 5 percent of AGI and AGl nvVr 
125^. The rate of over-claimers in this group was above average, but 
mucn lower than the previous one. Since awards wers lower to begin'with, 
the impact of misreporting was also lower. This group tended to increase 
the SEI even more after validation even though it was already high. In 
other words, over-claimers far outnumber under-claimers . The rates are 
not high enough to justify validation for all members of this group, but 
it would be well to keep an aye on results pertaining to it in the Task 2 
analysis. 

^noHEj6: independents who filed through BEOG or PHEAA with S£I = 
0. This group has a very low rate of niisreporting. Whatever errors are 
found in their applications, they are not ones that validation can 
identify. Perhaps quality control studies should check out for omissions 
on the part of these applicants, but for the time being no corrective 
action is needed. 

^^9^P V .' independents who filed through 3E0G or PHEAA with SgT 
greater than 0 . This group has a Urr- r •v;crtion missing with a 
moderately high rate of misreportinc those re-entering the system. 

The net result of this misreporting is by a slight margin to the 
applicant's disadvantage. Validation is probably not justified for this 
group. 

Qr°up .Sa : independents who filed through ACT or CSS, had SEI ° 0, 
ex_emptions not greate r than one^ were processed by April 30 and had AGI 
. n ot over S2,poq. This is another group with a low rate of misreporting. 
No corrective action is needed. 

Groue__29: j ndependents who filed through ACT or CSS, had SEI = 0, 
exemptions not greate r than one, were processed by April 30 and had AGI 
oyer S2,000 ,. This is a group very likely to have been affected by the 
SEI formula change that went into effect May 1. Their SEI under the new 
formula is zero, but their old SEI probably was not. Fully 75.5 percent 
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of them are missing in the nonvalidation sample, but validation seems to 
have actad as an incentive for them to remain in the system. No 
corrective action is needed, but the group should be observed in a year 
without an SEI formula, change. 

'^^°^P 30: independents who filed through ACT or GSS. SEI = 0 
exemptions not greate r than one, processed May 1 to July 31 . This is 
another low misreporting group for which no action is needed. 

SrouD_31: independents who filed through ACT o r CSS, SEI = 0, 
exemptions not greater t han one, processed after July 31. V.r.s group has 
lo'./ misreporting rates, but many more missing in the validation samples 
than in the nonvalidation sample. It is a good indicator that validation 
can delay re-entry. No corrective action is necessary. 

Group 32: independents who- filed through ACT or CSS, had SEI = 0. 
exemptions greater than 1 a nd year in school 1. or. blank . This is another 
low error group. No corrective action is needed. 

Group 33:. independents' who filed through ACT or CSS, had SEI = 0. 
exemptions greater than 1 and year in school greater than 1 . This group 
differs from the previous only in haying fewer missing cases.- No 
corrective action is needed.- 

'^ ''-°"P - ^ i- independents who filed through ACT or CSS, had SEI g reatpr 
than 0, were processed by A p ril 30. paid taxes not over S500. and signed 
their application. bv Februar y 28 or left the date blank . This group 
combines a high missing rate with high rates of misreporting. Since the 
group could have been partially affected by the SEI formula change, it is 
hard to judge the results. One hesitates to recormiend validation of this 
or the next two groups based" on observations from a year with an SEI 
formula change and a confusing systems generated transaction. Yet at the 
very least further study should be done. 

^y'^^P 35; independents who filed through ACT or CSS, had SEI greater 
than 0, were processed bv Ap ril 30. had taxes not over S500 and signed 
the application after February 23. This group had fewer misreporters 
than the previous one, but the rate is still higher than the average for 
independents. 
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■Qro"P.36' independents who filed through ACT, had SEI greater • 
than 0, were processed bv Ap ril 30. and had taxes over ';.qnn . over two 
thirds of these applicants failed to re-enter the system, but most of 
those who did were misreporters. Validation of just this group among 
independents might serve to obtain a better picture. However, results 
from this study could be confounded by the SEI formula change, a fact 
that makes suspicious any results separating applicants processed by 
April 30 (THAID was provided cut-off dat-s for the end of each month) 
from all others. On the other hand, the results might have had mora to 
do with the income tax filing. data than with the SEI formula. change. 

^ . ^P^^ ?l ' independents who filed through ACT or CSS, had SEI q reat.:»r 
than 0 and were processed aft^r A pril This group had fewer missing 
cases than the. preceding ones, but still a rate of misreporting higher 
than most independents. The rate is not high enough to warrant 
validation. 

Chapter 8 will coordinate the various comments into recommendations 
for use of the findings and will compare them with the PEC. 
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8 

RECOMMENDATIONS AND EFFECTIVENESS OF THE MODEL 



This chapter presents various recommendations for the use of the 
error-prone model, and compares its effectiveness with the PEC. The 
first section- presents a strategy for using the model to select 
applicants for validation. An alternative approach related to estimation 
of taxes is discussed in the second section, followed by a section on 
validation of independenlis . A comparison of the relative effectiveness 
of the EPM and the PEC follows. The chapter concludes with 
. recomnendations for further research and for program improvement. 
8,1 A Validation Selection Strategy in the Absence of Other Changes 

This section presents a strategy for selecting applicants for 
validation based on the model, assuming the model is the only information 
available and validation the only possible action to ^-^ taken. If OSFA 
makes dramatic changes- to the delivery system, the moael would not be 
valid. Similarly, if validation procedures are changed, the model 
likewise would have to be adjusted. The strategy simply consists of 
selecting all applicants from groups 2, 3, 4, 5, 7, 3, 10, and 24. These 
group: include 11 percent of all applicants in the random validation 
file. This combined group includes 23.3 percent over-claimers , ten 
percent under-cl aimers, and 26.2 percent missing. Of applicants who 
re-enter the system, 45.5 percent of members of these groups are 
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misreporters. The group would include 28.7 percent of all over-claimers 
(or at least of. those who filed by August 31). An additional one percent 
of applicants could be selected on a random basis to ensure continued 
investigation, for a total of 12 percent selected, accounting for 29.5 
percent of all over-claimers. 

As will be noted, no independents are included among the eight groups 
reconmended for selection. This is due to the much lower error rates for 
independents found in this study, which is probably somewhat due to the 
problem presented by the change in the SEI formula. 

This validation procedure wuld be more effective than the PEC taksn 
collectively. The PEC validation file has only 13.5 percent 
over-claimers and 29.2 percent of those who re-enter the systam 
misreporting. Some individual PEC are more effective, and the 
possibility of combining the two will be explored in a latsr section. 

. 8.2 Estimation of Taxes^ an Alternative Remedy . 

Seven out of the eight groups recommended for validation in the 
previous section had estimated tax figures. Estimated tax figures by 
. dependents accounts for over 30 percent of all misreporters. A good 
proportion of these errors could be eliminated through various policy 
changes. Among these are: 

• Instruct applicants to obtain a 1040/1040A form and complete it, 
obtaining help from a tax service if necessary, if they have not 
filed a tax return by the time they complete their applications 
and to sign a statement to that effect. 

•. Require that a copy of 1040/1040A returns be enclosed with 
application (possibly schedules filed with the 1040). 

• Require that financial aid administrators validate tax return 
only, for applicants who estimated taxes. 

• Conduct verification of income and taxes by mail for aoplicants 
who estimated (this, allows early filing, but requires 
corrections if error). 

• Provide a 1040 form and directions along with applications and 
instruct that it must be completed before the application is 
completed. 
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Implementation of one or more of these recommendations would probably 
reduce error considerably. The change in policy, however, would require 
a r<ivision of the model since estimation of taxes would no longer provide 
the information it implied in 1979-1980. Many misrsporters who may have 
used estimation as an excuse for over-claiming, might fall into other 
groups, but all in all, error would almost certainly be reduced. 
8.3 Validation of Independents 

Two issues related to misreporting patterns of independents need to 
be discussed.. First is the problem of the SHI formula change. This 
study must remain inconclusive as to the behavior of independents whose 
applications were first processed by May 8, 1979, due to circumstances 
which are not likely to recoccur in subsequent years. Thus, groups such 
as 34, 35, and 36 may or may not merit validation. The Task 2 analysis 
may be able to answer some of these questions,' since more complete data 
will be available for that analysis. 

The second issue is that the SEI is the best predictor of 
misreoorting (with MDE source predicting re-entry). Applicants with 
SEIO are not- found to be misreporting frequently through current 
validation procedures. This may be because the less the applicant 
reports, the fewer things the FAA is able to validate. In addition, it 
is very difficult to validate dependency status, and the most effective 
way for an applicant to misreport is to improperly claim independent 
status. Thus, If independent applicants were to be selected for 
validation under present validation procedures, groups 27 and 34 to 37 
would be selected. It would be less effective, however, to select these 
groups than to concentrate on the ten dependent groups mentioned in 
section 5.1. On the other hand, if a more effective way of validating 
low income applicants can be devised, then the present model should not 
be used to select applicants for that kind of validation. 
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8.4 The PEC Versus the EPM 

Various approaches were used to compare the effectiveness of the PEC 
with that of the model. One approach was to code members of the random 
validation group who wouM have met PEC by which PEC they met. A 
multiple linear regression with absolute award difference as the 
dependent variable and the EPM and PEC as predictors was conductad for 
aoplicants who re-entered the system. The EPM accounted for 4.3 percent 
of the variance of award difference. The PEC accounted for 1.5 percent 
of the variance. Combining the two accounted for 5.5 percent of the 
variance of the . dependent variable. The net result is that the EPM 
improves the PEC much more than the PEC improves the EPM. 

Looking at the PEC validation file, however, it became apoarsnt that 
some PEC were working well and others not at all. PEC that worked well 
were A6, AlO, All, A14, and A15. Each of these yielded over 20 percent 
over-claimers, with A6 working better than any group in the EPM {37% 
over-claimers). This, however, includes an over-representation of 
applicants which met other. PEC. Among applicants in the PEC file which 
met A6, 25 percent belongi to one of the eight groups suggested for 
selection in section 6'.1. PEC A6 involved a complex formula which was 
not used as a variable, so it does not appear directly in the model. The 
other PEC involved small percentages of the population and would not have 
appeared had they been included. 

It would probably be desirable to incorporate these five PEC into the 
selection criterion. They should be seen as an additional check along 
with the error-prone modeV. On the other hand, the PEC other than these 
five are probably unnecessary if the EPM is being used. 

8.5 Further Research Needed 

The present model is a first step which needs to be elaborated and 
refined. Its validity in subsequent years should be tested, as should 
its validity for applicants selected after August 31. The model is only 
appropriate for current • validation procedures, and should be refined as 
new methods of quality assurance are 'developed. 
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Quality control studies should investigats why apolicants make 
mistakes. Reasons should then be broken down by EPM group and 
appropriate edits generated or validation indicated when a pattern seems 
clear. 

Subsequent investigations may subdivide some groups and merge 
others. Policy changes such as requiring tax forms or inclusion of 
tighter edits may make the model obsolete. Matching the model with data 
obtained from the IRS analysis will be useful in obtaining "true" (vs. 
FAA-verified), income, as well as obtaining income for aoplicants 
classified as missing in this study. 

New variables presently available from the 1979-1980 data files 
should be tried, particularly with a view to breaking up the larger 
groups. Variables not presently available, but which could be acquired 
in a QC study should be used in developing new models, so that those 
which become successful predictors can be incorporated into future 
application forms. A top priority should be identifying those cases of 
serious misreporting that are scattered among groups with low error rates. 
8.6 General Recomrnendations and Alternatives Related to the EPM 

Several suggested improvements for the use of the results presented 
in Part 11 for quality assurance follow: 

Select groups 2, 3, 4, 5, 7, 8, 10, and 24 for validation 

Supplement EPM with PEC A6, AlO, AH, A14, and A15. 

Attack the error rate due to estimation. Several suggestions 
have been made in section 8.2. Consideration should be given to 
wider use of form 1040 for quality assurance. 

If independents are to be selected under present validation 
practices, select only those with SEI greater than 0. 

List an applicant's group number iri the SER. Provide school 
counselors and Financial Aid Administrators with profiles of 
each group (but not with validation criteria) as an aid to 
identifying possible sources of error and to helping 
under-cl aimers receive the amount they are entitled to. 
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CSrnCAL FIELD COMMENTS FOR iNONSUPPLEMENTAL APPLICANTS 
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300 4 9 2^6 

5 29 253 

302 5 50 254 

303 7 31 255 

306 11 32 257 

307 14 gg pep 

308 15 68 260 

309 17 59 295 

310 19 71 296 

311 20 72 

312 21 73 

313 22 75 

314 25 76 
3l3 25 77 
316 31 78 

36 79 

37 30 

38 31 
40 32 

107 34 
242 35 
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APPLICANT 13 T^JZCtS,^ 

Dependency Scaeua 

If aodel is ai^aing: " 

if side d^cr-e is - 'TD- :hen applicant is dependent: 
if sid^ .:-cr- is - ''E" then applicant i.^ independenc 

If side d-cr-5 is missing; 

applicanc depend^.'. independenc; ocher-.i^ 

Agi and Portions 

pfrcf^n.i ' .Ii adding the 2 

pcVjon'' ' ^S-- -in^ -.h. 1 

M Iz ,si U jr^MC «d both porticns bUnlc, a„«pe I portion., - 

>b?isr ir-:^?hinrLJ:!'""' - 

gnu sua I Extsenses fae) 

(b) if oi.her field i. a,issing.. che field pr..enc a. ue. " • 

CO if boch fields are aissiag, a„uae ue Co be ^ero. 
.Unreiabursed Tuiginn 

Aliasing, asstaae sero. 
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Assecs/Caah and SavVrn^ <. 

U any field is oi.3ing, assume zero for chac field. 
Taxes ?sid 

Mari=al St^cu^ and Household Size M^ssiag: 
Ca-l) ^ecendsncs: ,,,.,,1 ^.^^^^ , ^^^^^.^^ 

household si^Q - 3 

houaehald size - i 
(b; y..rUal Stacus Missing.TIousehold si.e. pre^.nc: 

Dependenc, hh-size g-aCar shan 2; 
assuae aarried 

Cb-2) Oepandenc, all other hh-si=e: 
assise divorcad 

V- J/ inciapendanc, hh-sisa ■ =L: 
assunia single . ' 

Cb-4) ladependenc, all other hh-sira: 
asstane aarried. 

Household SL£e Chh->sira) (Onlv if hK-,,-~. 

vith marital sLcusJ? " "'^'^ ^^^="2 

Ca) If hh-siie is .i3sing and ex.^pcions is present, .se e.^pcio.s. 
Cb) If exeaptions is aissing: 

^^^^ -actus is. 

tf applicant: i, d.p..da„t: and parents ar. deceased, assume 1. 
If i.idep<ir.dcrt and oarried, asswae 2. 

re independent and 3ingle/vidov.d/divor=ed/3e,..rared , as.uae I. 
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fa) rf amounc and aonchg are aisaing, assume zero. 

Co) U amount is'pr.sanc buc aonchs are a:issing, asau^a noaths ,o be 9. 

i-'S ^AJiiing ana aoncns is orssenc: 

ff ^^fi"*si2e is sero, assuae 249 
H {^^'•^i^^ ^5 one, assune 296 
if hh-sise Is ;:vo, assume 238 

(26 ciaes nuab^r in hh graat^r chan 2). 
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Prs-established Criteria 



PEC A: Applicants who 



P£C 3: 



PEC C: 



^. . establish eligibility after r^-'^^rtinn 



OT 3 transaction 
one or mors of the 



A-1: 

A-2: 

A-3: 

A-4: 

A-5: 

A-o: 

A-7: 

A-3: 

A-9: 

A- 10: 

A-U: 

A-12: 

A-13; 

A-14; 

A-15: 

A-1C\- 

A-17: 

A-13: 

A-19: 

A-20: 

A-21: 



Financial data iTiissing 

Financial data missing 

Applicant's resources reducad 

No S.S. match and applicant verifies 

NO S.S. match and applicant changes 

V.A. b ank or zero and applicant verifies 

/.A. blan< or zero and applicant changes 

V.A. greater than zero and less than S131 and aoplicant verifies 

Aj?iS?iJn".^i?S^"l^^^^ ^"^ ^^^^^'""^^ 
With 32 
with 
with 
with 



V.A. greater than 
Ccmbi nation of an 
Combination of an 
Combination oi^ an 
Combination of an 
Combination of an 
Combination of an 



criterion 
critarlon 
criterion 
criterion 
criterion with 02 



CI 
C2 
01 



B-l 
3-2 
3-3 
3-4 
3-5 
3-6 



Large EI change 
Larger EI change 

Combination of a 3 criterion with CI 
Combination of a 5 criterion with C2 
Combination of a 8 criterion with 01 
Combination of a 8 criterion with 02 



C-1; 
C-2: 
C-.3: 
C-4: 



Large £1 change 
Larger EI change 

Combination of a C criterion with 01 
comhi nation of a C criterion with 02 



PEC 0: Number in 



0-1: 
0-2: 
0-3; 



postsecondary education (PHE) and tax filing stat-js. 



PHE greater than 3 

Tax return figures are estimated 

Combination of 01 and 02 
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APPENDIX D 

GENERAL METHODOLOGY FOR ERROR-PRONE MODEL 
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Error-Prone Modeling Techniques 

The development of adequate error-p»cne models has become an 
important concern for various agencies. The need to determine which 
cases are most likely to be misreporting so that correction 
action-procedures can be instituted is likely to be present for any 
program engaged in the disbursement of public funds on the basis of 
stated need and qualifying conditions. 

Three major approaches to error-prone modeling have been used by 
various state and federal agencies. Each has its advantages and 
disadvantages and each is best suited to different kinds of situations. 

The first method, used by the Welfare Departments of South Carolina 
and the District of Columbia, uses discriminant analysis to obtain a 
fonnnla which assigns a score to each case. The higher the score, the 
more likely it would be that the applicant is misreporting. Thus if the 
agency wanted to select applicants most likely to misreport, it would 
simply select those applicants to whom the formula assigned the highest 
scores. 

The major drawback of this method is that it ordinarily assumes that 
a variable will affect all applicants in the same way. If, for exa(,iple, 
it turns out that estimating taxes is significant for dependent but not 
independent applicants, discriminant analysis will fail to take this into 
account. Thus it could easily fail to detect some important combintions 
of variables which could predict error-proneness. In addition, 
discriminant analysis would not point to specific areas where an 
applicant is likely to be misreporting. Since each applicant receives a 
single score one cannot distinguish those who misreported AGI from those 
misreporting rates paid. Of course, separate analyses could be conducted 
to predict misreporting for each specific field, but this method would 
lack parsimony and would be difficult to interpret. 

The second approach, used by the State of New Hampshire to identify 
error-prone cases in its MEDICAID program, attempts to define a single 
group most likeV to. .exhibit a high degree of misreporting. Depending on 
the. size of the group, every member or a certain percentage of this group 
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(and this group only) would be validated. Where only a small proportion 
of all the cases can be validated, and the principal objective is to 
maximize the savings in actual disbursements from the cases actually 
validated, this method can be very sffsctive. On the other hand, this 
approach is likely to overlook groups whose error rate might approach 
that of the selected group, and which might require a different type of 
corrective measure. The 3E0G program, with the use of edits in addition 
to validation, a high droo-out rate among applicants selected for 
validation . and the dual concern for deterrence as well as savings in 
disbursements to valiaated applicants, requit . a different approach. 

The third approach has been used by the State of West Virginia Aid to 
Families with Dependent Children program and by the Supplemental Secur:ty 
Income program of the Social Security Administration. It essentially 
divides the applicant population into mutually exclusive groups which 
differ as much as possible from each other in either the mean dicrepancy 
in expected disbursement or in the rate or type of misreporting. .rn's 
method has the potential to describe each 'group separately in terms of 
type of error, and thus to prescribe different types of corrective action 
for each. It has the further advantages of taking into account effects 
whichapply to only- part of the population, and of producing results 
which can be expressed in simple terms. This method, sometimes called 
.classification analysis, sequential structure search or automatic 
interaction detection (AID) is the one which will be used in this study. 

Overview of the Sequential Search Approach 

The term sequential structure seiirch is the more generic term for a 
conceptual method of exploratory analysis designed to discover nonlinear 
combinations among many variables which best predict a single dependent 
variable. The term Automatic Interaction Detector (AID) is often used 
synonymously with sequential structure search, but is often used more 
precisely to describe the implementation of this concept by the Institute 
of Social Research (ISR) at the University of Michigan. ISR has 
developed two programs which will conduct this type of analysis: AID3 
which accepts continuous dependent variables, and THAIO which accepts 
categorical deoendent variables. This software was used by the 
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SuDplemental Security Income program in their development of error-prone 
profiles. In addition, the West Virginia AFDC program used a software 
package closely related to THAID. In the subsequent discussion, the term 
AID will refer to the general technique rather than to a specific program 
or package. 

Techniques such as discriminant analysis or multiple linear 
regression make the assumption that a given effect will apply equally to 
all members of the population. The result of either of these techniques 
is an equation which is meant to predict the dependent variable for every 
member of the population. AID, on the other hand, does not make the 
assumption that a predictor variable will affect the dependent variable 
in the same way for all cases. Instead, AID starts by breaking up the 
sample into two subgroups selecting that split which produces groups that 
differ from each other on the dependent variable as much as possible, 
each subgroup is then split separately, allowing for different predictor 
variables to split different previously-formed subgroups. 

Interaction effects occur when a variable predicts differently for 
one group than for another. Ordinarily,- discriminant analysis and 
multiple linear regression do not take interaction effects into account. 
AID specifically identifies groups (using various combinations of 
variables) which will differ as much as possible on some criterion 
variable. Thus AID will be able to identify error-prone cases in 
instances where, for example, low taxes are an indication of 
error-proneness among high income applicants, but not for low income 
applicants. Linear models are oblivious to such relationships. 

AID accepts one dependent variable, which may be categorical (such as 
type of applicant) or quantitative (such as discrepancy in expected 
disbursement, expressed in dollars). Predictor variables can be 
monotonic (where the sample or any subgroup can only be subdivided into 
high and low groups based on some cut-off point) or free (used for 
categorical, as opposed to quantitative, predictors where any combination 
of values can be used to split the groups). In either case predictor 
variables must be coded in terms of a small number of possible values (no 
greater than 10 for THAID). 
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Out of ths many possible splits defined by predictors, AID selects 
the one which will divide the sample into two groups as different from 
each other as possible (the operational definition of "as different from 
each other as possible" is given in Chaoter 5). The process then 
continues for each of the two groups into which the original sample was 
split. When a group becomes very homogeneous, cannot be further split 
using the predictor variables available, or would yield subgroups under a 
certain size if it '.vere split, then the process is complete and it 
.becomes one of the groups defined by the model. If an applicant group is 
both large and heterogeneous, it would be an indication that some 
additional predictors should be sought and included in the analysis. 

Because AID investigates many possible combinations of variables, at 
times it produces results which are specific to a given samole. Two 
questions may be asked pertaining to the groups which emerge from an AID 
analysis: (a) Do the groups have the same characteristics in the 
population as they appear to have in the sample? and (b) do the groups 
constitute an optimal classification of the population if one is trying 
to predict the dependent variable? In order to answer either question 
one needs to use a second sample randomly drawn from the same 
population. The first question can be answered by checking whether the 
subgroups produced by AID from the first sample have similar 
characteristics in the second sample. The second question requires that 
a separate analysis be conducted for the second sample. It is quite 
possible that one would obtain a different solution if variables which 
are highly interrelated are used (this is similar to the problem of 
multicoll inearity in multiple linear regression). The question of 
whether a given solution is the best possible, however, is of secondary 
importance to whether the classification which emerges is effective in 
predicting error-proneness. 

The THAID Program 

While AID 3 is the most popular sequential search program, it does 
not handle nominal scale dependent variables which achieve more than two 
different values, THAID, on the other hand, is precisely designed to 
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handle nominal scale dependent variables which achieve up to ten 
different values. THAIO accepts up to 40 predictor variables, each of 
which can also achieve up to ten values. 

THAID, like AID3 splits each sample or subsample into two subgroups 
in such a way that the two subgroups are as different as possible in 
terms of the dependent variable. Two different statist ics—theta and 
delta— are available as operational definitions of "as different as 
possible." The theta statistic (from which the orogr.am gets its name) 
allows only splits where the two subgroups created by the split have 
different modes with respect to the dependent variable. In other wordSj 
the value of the dependent variable appearing most frequently in the 
first subgroup must be different from the value aopearing most frequently 
in the second subgroup. THAID selects at each step that split which 
maximizes the value of theta, where 

^heta^ ^fi^ 

Mj^ and represent the frequency of the mode in each subgroup (i.e., 
the number of cases attaining the value most frequently attained by the 
dependent variable in each subgroup) and and represent the 
total number of cases in each subgroup. 

Theta has the advantage of producing subgroups with different modes 
if the data allow it, an ideal situation when the purpose is selecting 
cases which achieve a given value on the dependent variable (in our case, 
applicants who are misreporting to their advantage). The disadvantage is 
that the data seldom allow it when one has a lopsided distribution, and 
in this study over-claimers and und3r-cl aimer: combined do not reach 15 
percent of the sample. 

The alternative is the delta statistic, which selects splits which 
are as different from each other as possible, taking into account every 
value of the dependent variable, not just the mode. When THAID splits 
thejsample or a subsample into two subgroups it calculates the delta 
statistic for each split. 
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Where 

Ni frequency in the first subgroup 

'^'2 ' frequency in the second subgroup 

Pj = proportion of cases in unsplit group failing on the j'^^ 
code of the dependent variable 

Pij = proportion of cases in i^'t split subgroup failing on 

the ji-n code of the dependent variable 

N = frequency in unsplit group 

. Nj = frequency in unsplit group in jth code 

G = number of possible values in the dependent variable. 

THAID choses.that split which can be defined by a predictor variable 
which max imizes delta. 

THAID produces four iterations, and therefore up to 16 groups can 
result from one run. However, one can also restrict the cases which will 
be included in a given run, so that a group which was defined by one run 
on the fourth iteration can become the entire sample for a subsequent 
run. Thus, in practice any number of iterations can be produced. 

The option to restrict the sample can also be used to force a first 
split or subsequent splits on theoretical grounds. This option was used 
to start off by splitting dependents from independents. Other forced 
sDlits "^re also tried, but not used in the final model. 

The output provided by THAID can include upon request information on 
the best split that could have been prod'jcad by each of the predictors at 
each step, anf^ the value of theta or delta for each predictor in each 
instance, this information can be used to prepare alternate runs with 
forced splits or to decide which predictors to delete in subsequent runs. 
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other options available in the THAIO program Include: 

• a weight function which can be used with a stratified sample or 
if one wishes a value of th- jc'DOfrit^nt variable to appear to be 
the rnode if it exceeds a ceru^iir: ^'-e-i^r.': icje of the cases. 

• an option to limit the minimum size a group ..an duhicvi^ 

• an option to set a mi.nimum value for theta or delta before a 
split can be produced. 

Limitations of the Approach 

One of the major linnitations of this aoproach to error-orone modeling 
IS that one can never be .certain of having produced the best possible 
solution. A different investigator, using different samples, different 
variables, or forcing a different first split could well come up with a 
totally different model. What could have been an effective split at the 
second iteration may never appear in light of a different split at the 
first. 

THAID has a tendency to select splits that come close to dividing a 
sample or subsamole in an uneven division. For this reason criteria 
which define only a very Sinall oercentage of the pooulation would be 
unlikely to appear in the model. The one tenth of one percent of the 
applicants defined by a given PEC could be expected t^ be scattered among 
the thirty^seven groups resulting from the study. For this reason the 
model should not be used to the exclusion of other criteria that have 
also proven effective. 

Limitations of the study (as opposed to the THAID method in general) 
will be discussed in Chapter 6. Chapter 3 will discuss the specific way 
in which THAID was applied In the present investigation. 
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This appendix describes the sequential search procedure used in 
developing the model. Each derived model and subsequent refinement is 
brief Ij^ explained. The very first THAIO runs were conducted on the 
working sample using only variables which could be obtained directly from 
the.SER, plus the variable "number of prior transactions". While these 
runs were performed on dependents and independents separately, the 
independent runs producad erroneous results since the systems generated 
transaction and the change in EI formula had not been accounted for. 

These runs used the delta statistic placed at ICO the oiinimum 
membership a group was allowed to have. After the first run produced 
sixteen groups (for dependents only) each group was in turn submitted to 
further iterations. The cross-validation attempt using the replication 
sample revealed some of the splits to be idiosyncratic (i.e., specific to 
the sample), and in the end twenty groups were retained. 

A split was considered to be replicated if a similar :;ivision was 
obtained .in the replication sample; this was determined by comparing the 
resulting four subgroups— two in the .-.orking sample with two in the 
replication sample. In other' words, if Wl and W2 are the two subgroups 
resulting from a split in the working sample and Rl and R2 the 
corresponding groups in the replication sample, 'fll must be more similar 
to Rl than to W2 or R2, and so on for the four groups. The decision on 
similarity, however, was made on an intuitive basis, and some legitimate, 
but trivial, splits. were at times discarded if the resulting groups were 
small. 

The very first model, derived exclusively for dependent applicants, 
was a reasonably adequate model except for a few conceptually 
inappropriate characteristics. One of the defining variables was 
non-taxable income other than Social Security. The program separated 
those who left this field blank from those who did not, but in doing so 
it was merely selecting PHEAA applicants (who did not have a place to 
include this field in their applications) and 8E0G applicants who left 
the field blank Vom the rest of the SEOG applicants. 
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A saconc inadequacy of the model was that taxes paid as a proportion 
of AGI was coded as a free variable, a THAID option wh -^h allows splits 
other than "high" vs. "low." This was very effective since the group 
with the (Tiiddle range is less likely to misreport than those at the two 
extremes. The problem is that some subgroup consisted primarily of 
members of one extreme (low taxes with respect to AGI. for example) with 
a few strac,jlers from the other extreme. This made the definition 
.:.::tually awkward and reduced the homogeneity of the groups. 

.ariables which were not effective at any step of the process or 
which if erfective failed to replicate were removed. The coding for some 
variables was changed. NTI replaced Social Security Benefits and 
Non-taxable-other, Taxes as a proportion of AGI was refined and changed 
so as to allow only "high" vs. "low" splits. New ratios were created in 
view of the success of "taxes as a proportion of AGI". Corrections 
related variables were also created. 

At this point a number of experimental runs were conducted using only 
data from the dependent applicants. Variables were added, and in some 
cases subsequently eliminated.- Splits based on theoretical s/.pectation 
were forced on the data. The theta statistic was used. Cases were 
weighred based on their score on the dependent variable (TYPE) so as to 
increase the probabi lity of finding small groups with large percentages 
of over-claimers and under-claimers, and analyzed using both the theta 
and the delta statictic. The roles of the working and replication 
samples were switched. 

It soon became apparent that theta as a splitting criterion was not 
appropriate regardless of how one weighted the data. It also became 
apparent that three variables were competing for a role at the very first 
split of the dependent subsamples. Very slight changes in the data or 
the procedures allowed one or another of these variables to create the 
first split, and depending on which variable was chosen by the program, 
the subsequent splits produced very different results. These three 
variables were tax filing status (separating those who estimated from all 
others, MOE. source (separating BEOG and PHEAA applicants from ACT and 
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cr,S; and SEI (with various cutoff points at ti;a lower and of the 
scgctrum). When no weights were used, ths first two j/' these variables 
p. oduced very similar deltas. 

"-^iien tax filing status was used as the first splitting variable, MOE 
sourca proved effective for those who did not estimate taxes (used '::ax 
form, did not have to file or did not answer th« question). When .MDE 
source was selected to create the first split, tax filing status was 
effective for ACT r^nd CSS applicants, but not for 3E0G and PHEAA. 
Eligibility index appeared frequantly, but not first, in the unweighted 
runs, while often creating the first split with certain combinations of 
weights. Weighted runs often discovered small groups with high 
percentages of misreporters, but were also more likely to yield results 
which did not replicate. 

Con- '^t ions variables in general did not predict well. This was 
thought to be because it is difficult to define corrections variables 
which apply to a- large proportion of applicants and are at the same time 
affective. The desirability of isolating cases where corrections history 
might make a difference became apparent. 

Forty variables were identified as offering the best possible chances 
for success.. Three diff erf ent procedures were imp lamented 
simultaneously, using the working sample. The first procedure was a 
simple, unweighted THAID analysis, forcing the first split on dependency 
status, using the delta statistic and setting at 50 the minimum number of 
cases in any group resulting ^'rom a split. The second was identical, 
except that a second forced split, separating first transactions from 
those with prior transactions was implemented. The third procedure used 
a weighting scheme designed to produce equal weighted frequencies for ail 
four applicants types for each dependency status. Since group size had 
to be determined using weighted frequencies, they were sat so group sizes 
would average 100, but could be smaller for groups with large proportions 
of misreporters. The delta statistic was aUo used here. 

The simple unweighted model proved to be the most effective, after 
incorporation of a few additional divisions suggested by th-a early 
model. The forced split on prior transactions model yielded more groups, 
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and did use some of the corrections variables. It also left some large 
groups unsplit to a greater extent than the simple model, did not 
identify groups with resLlts as extreme as tlios'3 of the simple model, and 
was less parsimonious. The weighted model identified a few more extreme 
groups, but left many large groups unsplit. 

The simple model was selected and further investigated. Proportions 
of misreporters for nonvali dation applicants were calculated for each 
group. Mean values of the critical fields, SEI and scheduled award on 
the criterion and selection transactions were calculated for each grouo 
using the working sample (computed witn and without absolute values). 
Percentage of non-missing applicants correcting each critical field in 
each direction were also calculated for each group. Thus a profile of 
each of the groups djfined by the model was obtained. 
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GLOSSARY OF TERMS 



ACT: 



AGI: 



AR: 



Assumption 
Edit: 



BEOG: 



Comment: 



Composite 
Index: 



Correction: 



Critical 
fields: 



American Colleqe Testing Program, ACT's Family Financial 
Statement (FFS) is one of four applications with which a 
student niay apply for a Basic Grant. (See i^DE for a list 
of other sources.) 

Adjusted gross income- AGI is an aoplication item which 
refers to wages, salaries, tips, farm income, dividends, 
interest, and business income, as reported to the Internal 
Revenue Service. 

Applicant's resources. AR is an application item which 
refers to the total of the dependent applicant's -and 
spouse's) cash; savings and checking accounts, and other 
assets, 

One t.ype of the processing system's series of 
computerized edits designed to check for missing 
information, as well as the logic and consistency of all 
data provided. With this edit, the processing system 
assumes a value for a missing or apparently incorrect 
application item based on other provided data. 

Basic Educational Opportunity Grant, BEOG's application 
form is one of four forms with which a student may file for 
a Basic Grant. (See MDE for a list of the other sources.) 

An instructional messaga on the SER, Many comnents ^-.^-e 
the student to review the application and ta':: ''urth^r 
action, if necessary. The computerized comrniirt.. ar': 
triggered by "he processing system ed7:ts, 

An indicator used in this study to measure api. : ant 
corrections behavior. T* .:nvo.ite index is calculated by 
multiplying the percenta^/* '-c-pl icants making corrections 
by the effective SEI cha/z^- 

Following the initial api>: \ ':ion, a student may change any 
or all of his or her appT.ij-.-.ion inforrriation, A correction 
may be in response to an edit or the validation process, or 
it may be done at the student's initiative. For analytical 
purposes, corrections are divided into ones occur ing pre- 
and post-selection/eligibi lity. 

The 14 application items used in tnis zt:i4y. These items 
are: AGI, AR, HS, Model, MS, NA, NTI ..it:. Portions, Tax 
Filing Status, TP, UE, UT, and VEB, 
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College Scholarship Service. CSS's Financial Aic 'form 
(FAF) is one of four applications with which a student may 
apply for a Basic Grant. (See MDE for a list of other 
sources.) • 

One feature of the application processing system designed 
to minimize the number of student eligibility 
determinations made on the basis of invalid, inaccurate or 
incomplete data. The edits screen the applications by 
checking for missing information and the logic and 
consistency of all data provided. 

The actual point change resulting from a correction if SEIs 
above 1600 are set at 1600. For sxamole, the absolute 
change for an applicant whose SEI goes from 500 to 2200 is 
1700; however, the effective change is 1100, or 1500 minus 
300. 

The 'status of an applicant who is qualified to receive a 
Basic Grant. Eligible applicants have an SEI from 0 to 
.1600. 

error-Prone Model, The result of a procedure to identify 
applicants who are likely to nisreport on their 3asic Grant 
applications . 

The amount of award an applicant is due to receive, taking 
' nto account enrollment status, cost of education at the 
institution indicated on the application, and SEI. The 
applicant's award is referred to as "expected" because the 
data on the recipient file is not validated until after the 
end of the academic year. However, it is expected that 
expected disbursements are reasonable predictors for actual 
disbursements. 



Refers to applicants appearing of the rer-pient fiU. 
Financial Aid Administrator. 

Household size. HS is an application item which refers to 
the number in the household for whom the parents 
(dependents) or the applicant (independents) planned rc 
provide more *"^an half the support during the 1979-80 
academic year. 

The status of an applicant with an SEI greater than 1600 
who is not qualified to receive a Basic Grant. 
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The dependency status— indeoendent or dependent— of the, 
applicant. An applicant is a dependent if he or she 
answered "yes" to one or more of the following application 
questions: 

Did or will the student live with the parents for six 
weeks or more in 1978, 1979, or 1980? 

Did or win the parents claim the student as a tax 
exemption in 1973, 1979, or 1980? 

Did or will the student receive $750 or more worth of 
assistance from the parents in 1978, 1979 or 1980? 

Marital status. MS is an application item which refers to 
whether the applicant (independents) or the parents 
(dependents) are single, ^narried, divorced, separated or 
widowed. 

Multiple Data Entry. Process by which an individudl can 
apply for a Basic Grant using any one of the ro' lowing four 
application forms: 

• CSS's Financial Aid Form (FAF) 

• ACT'S Family Financial Statement (FFS) 
« PHEAA's application 

• 3E0G application 

Net assets. MA is calculated from several aoplication 
Items. It equals the sum of the estimated market value of 
the applicant's (independent) or parents' (dependent) home, 
real estate, investments, business, farm, and checking and 
savings accounts minus the liabilities on these items. 

Applicants who were not selected by the processing system 
to present certain documents to their financial aid 
administrators which confirm the accuracy of the 
information on their application form. 



Refers to applicants not on the recipient file. 

Nontaxable income. NTI is the sum of two application items 
which refer to the amount of social security benefits, 
child support, welfare, unemployment compensation, 
veteran's benefits (excluding veteran's educational 
benefits) and other income not gained through employment 
and not subject to U.S. income tax. 
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PEC: 



PHE: . 



PHEAA: 



Portions ; 



;i1gibility: 



Pre-/Post- 
Selection: 



Random; 



Rejected: 



Rejection 
edit: 



Pre-established criteria. The PEC are standards used to 
identify for validation those students most likely to 
misreport information on their application. The criteria, 
based on prior analyses of misreporters, s-^t up categories 
of applicants with questionable informati ve on their 
application or suspicious corrections behavior, and target 
a certain number of applicants in each category for 
validation. (See Appendix D) 

An application item which refers to the number in the 
applicant's (independent) or parents' (dependent) household 
which planned to be enrolled at least half-time in a 
Dostsecondary educational institution during the ig/g-l^SO 
academic year. 

Pennsylvania Higher Education Assistance Agency. PHEAA's 
application form is one of four forms with which a student 
may file for a Basic Grant. (See MDE for a list of the 
other sources.) 

The sum of two application items which refer to the income 
earned through employment (wages, salaries and tips) of 
both the applicant and spouse (independent) or the mother 
and father (dependent) . 

Refers to specific transactions of nonval idation apolicants. 
Post-Qligibi lity transactions are subsequent to the 
.tr? saction when, the aoplicant achieved eligible stdtus. 
Pre-dligibility transactions include both the transactions 
prior to eligibility and the transaction when the aDPlicant 
achieved eligible status. 

Refers to specif ic transactions of validation aoplicants. 
Post-selection transactions are subsequent to the 
transaction when the applicant is selected for validation. 
Pre-selection transactions includes the selection 
transaction and all previous transactions. 

Refers to the small portion of validation applicants that 
were not selected for validation as a result of 
pre-established criteria. 

The status of an applicant for !/hom an eligibility 
determination cannot be made because of missing information 
or suspiciously inconsistent application items. 

The most restrictive of the processing system's computerized 
edits. Following the receipt of a rejection edit, the 
apolicant must provide missing information or verify or 
correct existing data before the SEI can be generated. 
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Reject 
Reason Code: 

SEI: 



SER; 



Solicited 
Correction: 



Suspect 
Fields: 



Tax Filing 
Status: 



rP: 
UE: 



Each of the 43 reject edits is identified v/ith a code from 
. alpha A to I to numeric 1 to 17. 

Student Eligibility Index. A numeric indicator of the 
apolicant's financial strength based on information 
provided on the application- When the SEl is in the 
eligible range— from 0 to 1600--it is combined vith the 
applicant's educational cost to determine the size of the 
applicant's grant. 

Student Eligibility Report, The SER is the official ..lode 
of communication between the processing center and the 
aoplicant. The SER notifies the applicant about his or her 
eligibility status and carries comouterized messages about 
the application. The student must submit a signed SER to 
his or her financial aid administrator before a Basic Grant 
can be awarded. 

Refers to a change of application information on the 
transaction just subsequent to receiving a processina 
system edit. 

Refers to the application, items which caused the applicant 
to be selected from validation. All of the PEC Group A 
sub-criteria were designed to elicit a correction to one or 
more fields. 

Refers to the application item which asks applicants if 
(1) tax return figures are based on a completed return, (2) 
if those figures are, estimated, or (3) -if a tax return will 
not be filed for 1978. 

Taxes paid, TP is an application item which refers to the 
amount of Federal Income Tax paid for 197S. 

Unusual expenses, UE is the sum of two aoplication items: 
(1) the amount of muJ-cal and dental expenses not covered 
by insurance plus (2) the amount of loss due to theft, or 
damug^ by fire, stonr^ or accident not covered by insurance 
for ].978. 



UT: ' Unreimbursed tuition. UT is an application item which 

refers to the amount of elementary, junior high, or high 
school tuition or fees paid for dependents in the household 
during 1978. 

Unsolicited Refers to a change of application information that is not 
Correction; in response to a processing system edit* 



187 



Applicants who were selected by the orocessinq system to 
present certain documents to their financial aid 
administrators which confirm the accuracy of the 
information on their application form. 

Veteran's educational benefits. V£3 is an aoplication itam 
which refers to the monthly amount of educational benefits 
that the applicant expected to receive from the Veteran's 
Administration during the 1979-L980 academic year. 
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